Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Sun, 2024-10-13 at 16:26 +0100, Richard Purdie via
lists.openembedded.org wrote:
> I've spent quite a bit of time trying to reproduce/debug this:
> 
> https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
> 
> 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_apt: FAILED (218.02s)
> 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
> 
> on debian11-vk-1.
> 
> I've a successful test and a failed test testimage output for
> comparison:
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465
> 
> along with qemu serial output:
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106
> 
> I've tried rerunning that exact series of tests on that worker with
> no
> success reproducing the issue. The image was overwritten by a later
> test so we can't retest the exact image. I've checked the journal on
> that system and there isn't anything related around the time this
> happened. The two failures were on different network interfaces and
> both interfaces work on later tests.

This is happening on most builds on debian11 for my test branch. If I
add new changes, the issue doesn't happen so it looks to be timing
related and needs a populated sstate cache.

I put auditing into runqemu's tap locks codepaths and the devices are
being locked/released correctly, there is no duplicate device usage.

I also went through the non-tap/tun codepaths and I can't spot any
issues, we do use slirp in some tests.

Since we have a trigger point where we know it is failing (the runtime
ping and ssh tests), I added a os.system("ps awx") into it. That gave
me a process dump of what was running when this happens.

https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3603268
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3902903
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.4042146

From those we can conclude that there is only one qemu-system-* running
using the appropriate interface. I couldn't spot any other particularly
untoward processes running. That tells us a lot of things it isn't I
guess.

In addition to the ps, I've now added a netstat -tunape to see if we
can see something holding a network connection open. I'm wondering
about our httpserver used in some tests, or debuginfod, or something.
That build is ongoing.

> I'm open to ideas...

I still am, this one is proving to be a pain...

Cheers,

Richard