Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Sat, 2024-10-19 at 12:05 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Sun, 2024-10-13 at 16:26 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > I've spent quite a bit of time trying to reproduce/debug this:
> > 
> > https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
> > 
> > 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> > runtime_test.TestImage.test_testimage_apt: FAILED (218.02s)
> > 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> > runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
> > 
> > on debian11-vk-1.
> > 
> > I've a successful test and a failed test testimage output for
> > comparison:
> > 
> > https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
> > https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465
> > 
> > along with qemu serial output:
> > 
> > https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
> > https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106
> > 
> > I've tried rerunning that exact series of tests on that worker with
> > no
> > success reproducing the issue. The image was overwritten by a later
> > test so we can't retest the exact image. I've checked the journal
> > on
> > that system and there isn't anything related around the time this
> > happened. The two failures were on different network interfaces and
> > both interfaces work on later tests.
> 
> This is happening on most builds on debian11 for my test branch. If I
> add new changes, the issue doesn't happen so it looks to be timing
> related and needs a populated sstate cache.
> 
> I put auditing into runqemu's tap locks codepaths and the devices are
> being locked/released correctly, there is no duplicate device usage.
> 
> I also went through the non-tap/tun codepaths and I can't spot any
> issues, we do use slirp in some tests.
> 
> Since we have a trigger point where we know it is failing (the
> runtime
> ping and ssh tests), I added a os.system("ps awx") into it. That gave
> me a process dump of what was running when this happens.
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3603268
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3902903
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.4042146
> 
> From those we can conclude that there is only one qemu-system-*
> running
> using the appropriate interface. I couldn't spot any other
> particularly
> untoward processes running. That tells us a lot of things it isn't I
> guess.
> 
> In addition to the ps, I've now added a netstat -tunape to see if we
> can see something holding a network connection open. I'm wondering
> about our httpserver used in some tests, or debuginfod, or something.
> That build is ongoing.

https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/5/log.do_testimage.123265

This has netstat from the host but also uses the serial connection to
run ping on the host/server IPs and ifconfig.

This tells is the interfaces have the addresses we expect on both sides
and that packets are tx'd but not rx'd on the other side.

Creative ideas on further debugging welcome, I'm not sure where from
here. I think it narrows down the issue to the host kernel or qemu and
these qemu binaries are working on the other builders. The host is a
5.10.223 kernel:

$ cat /proc/version 
Linux version 5.10.0-32-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)

Cheers,

Richard