Progress on AB-INT issues // Help needed!

[Paul Barker <paul@pbarker.dev>]

[-- Attachment #1: Type: text/plain, Size: 3257 bytes --]

Hi all,

Firstly I want to thank everyone who has helped out with the
intermittent autobuilder issues and our current high priority issues so
far. There have been contributions from several different folks and they
are really appreciated.

This help means that we've made good progress!

We've been able to resolve several AB-INT issues so far (#16118, #16121,
#15912, #15188, #15495) and have patches available for others which need
reviewing and merging (#15696, #15885). Some issues are resolved on
master and just waiting on backporting of patches to the relevant stable
branches (#16000, maybe others).

We need to keep up the momentum as we still have a couple of open issues
blocking the Yocto 6.0 M1 build and we still have too many intermittent
failures occurring on the autobuilder.

There are a few issues I can flag where we need extra help:

- segfault in matchbox-windowmanager on qemux86-64 intermittently
  (https://bugzilla.yoctoproject.org/show_bug.cgi?id=13908):
  It's unclear why the simple-looking code here is segfaulting. Ross has
  tried running with ASAN but that didn't give us any new information.
  Can someone add further debugging to the code to catch what's going
  on?

- AB-INT: fetch/sstate/CDN - Missing objects in the cache: Bad file
  descriptor (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15945):
  The error here suggests we may have exceeded limits on the number of
  open file descriptors. Can someone add diagnostics to the relevant
  code path to print how many open file descriptors we have when the
  error occurs?

- AB-INT: runqemu.QemuTest.test_qemu_can_shutdown hangs with "Data
  received serial thread"
  (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15698):
  This needs further investigation.

- AB-INT PTEST: openssl ['Dubious,_test_returned_1', 'openssl'] test
  failure (https://bugzilla.yoctoproject.org/show_bug.cgi?id=15357):
  We need someone to check the logs and confirm that it is always the
  same test case that is failing. If that's true, we can take a patch to
  skip the flaky test.

- Various gstreamer PTEST failures (#16127, #16108, #15408, #15973,
  #15910, #15718):
  Each of these needs to be reviewed to see if upstream have replied to
  our bug reports. If any haven't been filed upstream then they need to
  be. If there isn't an upstream fix available, we can take patches to
  skip the flaky tests.

- Various PTEST failures on RISC-V (#15895, #15884, #15942, #15929,
  #15896):
  We don't use KVM for running qemuriscv64 so tests can be much slower,
  this may be causing various timeouts. Some of the failing cases create
  a large amount of data so there may be issues with buffering. Further
  investigation is needed, we could take patches to skip these tests on
  RISC-V.

We will have another call tomorrow at 09:00 PST / 17:00 UTC to discuss
the issues further and make sure we're focusing effort in the right
places. Please attend via zoom if you're able to help out:
https://lists.yoctoproject.org/g/yocto/topic/yocto_project_ab_int_bug/117364044.

If you're unable to attend the call then you can still help by picking
up one of the issues above.

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]