Re: [OE-core] Latest AB-INT unexplained mystery failure

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

It was suggested I should write down a list of other issues we have on
the autobuilder right now. Some of these are related to the transition
to new infrastructure, some are previously seen issues occurring more
frequently, some are entirely new. I appreciate these should all have
bugs. I'm copying various people including those helping with SWAT so
we can transition those we don't get solved into bugzilla.

a) insane do_package_qa intermittent failures
---------------------------------------------

Two levels of probems. The cachedpath code returns False instead of
exceptions so there was unexpected API changes. The recent insane
changes also stopped handling DEBIAN files correctly leading to races.
This is perhaps the best backtrace we have:

https://valkyrie.yoctoproject.org//#/builders/43/builds/246

Who/Plan: RP has a partial revert for the first part to fix builds.
Ross is aware and working on the second.

b) docs build failing in missing inkscape
-----------------------------------------

e.g. https://valkyrie.yoctoproject.org//#/builders/34/builds/9

We could install inkscape on all the workers but I'm reluctant to do so
as it starts to add large dependency chains potentially and reduces our
host dependency checking. We could to with other tools to generate pdf
and epub docs during release and docs builds too.

Proposal is to add a new buildtools-docs target to the AB which can
pull in the meta-oe layer and build a more fully features docs
buildtools. This would also be useful for the screenshot QA imagemagick
tests we want to add.

Who/Plan: TBD

c) source mirroring failing on AB
---------------------------------

https://valkyrie.yoctoproject.org//#/builders/82/builds/15
https://valkyrie.yoctoproject.org//#/builders/82/builds/16

Sources are still mirrored off typhoon but we've stopped that cluster
and are running off valkyrie. This could mean sources don't appear
"fast" in the mirror until that mirroring moves to the new NAS.

Who/Plan: Michael to transition mirroring to run of valkyrie NAS

d) CDN artefacts are failing

Same deal as sources, we need to move the mirroring to be based off
valkyrie's NAS.

Who/Plan: Michael to transition CDN to run of valkyrie NAS

e) bitbake server timeout issues
--------------------------------

https://valkyrie.yoctoproject.org//#/builders/48/builds/185/steps/14/logs/stdio

I've saved 

https://valkyrie.yocto.io/pub/shared-failure-data/fedora41-vk-1-selftest/

 - the json logs are something we've not had before for that
 - the key message in cookerdaemon is "Idle loop didn't finish queued
commands after 30s, exiting."
 - that message is there twice, it failed twice
 - can we decode the json logs and get timestamps
 - looks like it happens in particular in the siggen code?

Who/Plan: TBD

f) CVE database corruption
--------------------------

See list discussion:

https://lists.openembedded.org/g/openembedded-core/message/205715

Has a bug:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14899

I'm at a loss on this one. Started to wonder if an rsync job is
trampling the file. Check with Michael.

Who/Plan: Ask Michael about rsync job

g) Toaster test issues
----------------------

Toaster testing is more intermittent on the new faster workers. Have
some patches in progress but is has highlighted issues with the tests.

Help in changing "assertTrue(X in Y)" to "assertIn(X, Y)" in the
toaster tests would be welcome. I've made a few patches, more are
needed.

Help in being able to delete a project from the database before
starting tests would also be useful. Having trouble doing it in the
tests themselves due to database locking and can't work out the way to
call the rest DELETE API from selenium yet.

My plan is to length the timeouts and drop all the sleep/poll calls,
clean up the timeouts. This will make the tests much faster too. This
does mean adding "wait for alert to display" code. Have tried doing
this but tests need fixing.

Who/Plan: RP has ideas but would welcome help. RP needs to send
"debugging toaster tests email with tips".

h) weird networking issues causing test failures

See original email in this thread:
https://lists.openembedded.org/g/openembedded-core/message/205723

RP is at a loss.

Who/Plan: TBD

i) rust toolchain test failures mips/ppc

e.g.: https://valkyrie.yoctoproject.org//#/builders/21/builds/226

Who/Plan: Need to write email asking for mips/ppc help. Can
Adrian/Chuck help?

j) SPDX build warnings

RP hasn't merged Joshua's patch. Need to include in next testing run

Who/Plan: RP to test and merge patch

k) ssh test still causing failures

RP's fix looks to be incorrect, changed the wrong number. Correct fix
queued in master-next

Who/Plan: RP to test and merge patch


I'm sure there are more but I've put the ones I have in my head down
for now. I'm pretty sure people find fixing one issue painful, trying
to keep track of this many in my head is bad enough without trying to
fix them! Help on any of these is welcome.

Cheers,

Richard

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Ross Burton]

On 14 Oct 2024, at 14:46, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> It was suggested I should write down a list of other issues we have on
> the autobuilder right now. Some of these are related to the transition
> to new infrastructure, some are previously seen issues occurring more
> frequently, some are entirely new. I appreciate these should all have
> bugs. I'm copying various people including those helping with SWAT so
> we can transition those we don't get solved into bugzilla.

I thought I’d send an update to this mail:

> a) insane do_package_qa intermittent failures
> ——————————————————————

This has now been fixed.

> b) docs build failing in missing inkscape
> -----------------------------------------

This was the AB building more than we expected, and is now resolved.

> c) source mirroring failing on AB
> ————————————————

This is still in progress.

> d) CDN artefacts are failing

This is still in progress.

> e) bitbake server timeout issues
> --------------------------------

Still TDB.

> f) CVE database corruption
> --------------------------

Prognosis is good, appears to be fixed in master.

> g) Toaster test issues
> ----------------------

RP has discovered his inner web developer and is making good progress on these.

> i) rust toolchain test failures mips/ppc

These are proving tricky to reproduce on demand.  As the platforms are not supported upstream I think we should skip the rust tests entirely.

> j) SPDX build warnings

Appears to be resolved with Josh’s patch.

> k) ssh test still causing failures

Fixed.

Cheers,
Ross

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

On Sun, 2024-10-13 at 16:26 +0100, Richard Purdie via
lists.openembedded.org wrote:
> I've spent quite a bit of time trying to reproduce/debug this:
> 
> https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
> 
> 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_apt: FAILED (218.02s)
> 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
> 
> on debian11-vk-1.
> 
> I've a successful test and a failed test testimage output for
> comparison:
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465
> 
> along with qemu serial output:
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106
> 
> I've tried rerunning that exact series of tests on that worker with
> no
> success reproducing the issue. The image was overwritten by a later
> test so we can't retest the exact image. I've checked the journal on
> that system and there isn't anything related around the time this
> happened. The two failures were on different network interfaces and
> both interfaces work on later tests.
> 
> I'm open to ideas...

https://valkyrie.yoctoproject.org//#/builders/35/builds/229/steps/14/logs/stdio

debian11-vk-2 this time. A different runtime test, same kind of network
ing glitch.

Does debian11 have some kind of network management that is interfering?

Cheers,

Richard

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

On Mon, 2024-10-14 at 14:46 +0100, Richard Purdie via lists.openembedded.org wrote:
> c) source mirroring failing on AB
> ---------------------------------
> 
> https://valkyrie.yoctoproject.org//#/builders/82/builds/15
> https://valkyrie.yoctoproject.org//#/builders/82/builds/16
> 
> Sources are still mirrored off typhoon but we've stopped that cluster
> and are running off valkyrie. This could mean sources don't appear
> "fast" in the mirror until that mirroring moves to the new NAS.
> 
> Who/Plan: Michael to transition mirroring to run of valkyrie NAS

I talked with Michael and there is no automated rsync. Software has
been updated since the last one and that is why things are not
updating. As the mirroring is transitioned this issue will go away.

> f) CVE database corruption
> --------------------------
> 
> See list discussion:
> 
> https://lists.openembedded.org/g/openembedded-core/message/205715
> 
> Has a bug:
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14899
> 
> I'm at a loss on this one. Started to wonder if an rsync job is
> trampling the file. Check with Michael.
> 
> Who/Plan: Ask Michael about rsync job

There is no issue with any rsync job. I did realise I'd not merged the
patch I thought I had though. I will do so and see if that isolates the
issue.

Cheers,

Richard

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

On Mon, 2024-10-14 at 14:46 +0100, Richard Purdie via
lists.openembedded.org wrote:
> e) bitbake server timeout issues
> --------------------------------
> 
> https://valkyrie.yoctoproject.org//#/builders/48/builds/185/steps/14/logs/stdio
> 
> I've saved 
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/fedora41-vk-1-selftest/
> 
>  - the json logs are something we've not had before for that
>  - the key message in cookerdaemon is "Idle loop didn't finish queued
> commands after 30s, exiting."
>  - that message is there twice, it failed twice
>  - can we decode the json logs and get timestamps
>  - looks like it happens in particular in the siggen code?
> 
> Who/Plan: TBD
> 

Happened again:

https://valkyrie.yoctoproject.org/#/builders/48/builds/192

fedora41 again. Logs into:

https://valkyrie.yocto.io/pub/shared-failure-data/fedora41-vk-1-selftest/2

Cheers,

Richard

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

l) libz reproducibiity failure

https://valkyrie.yoctoproject.org/#/builders/37/builds/265
https://valkyrie.yoctoproject.org/#/builders/37/builds/266

https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241014-68a7dqot/packages/diff-html/

This looks familiar somehow. Looks like a 'configure' issue of some
kind (it isn't autoconf).

Who/Plan: TBD

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

On Mon, 2024-10-14 at 14:46 +0100, Richard Purdie via
lists.openembedded.org wrote:
> It was suggested I should write down a list of other issues we have
> on
> the autobuilder right now. Some of these are related to the
> transition
> to new infrastructure, some are previously seen issues occurring more
> frequently, some are entirely new. I appreciate these should all have
> bugs. I'm copying various people including those helping with SWAT so
> we can transition those we don't get solved into bugzilla.
> 
> a) insane do_package_qa intermittent failures
> ---------------------------------------------
> 
> Two levels of probems. The cachedpath code returns False instead of
> exceptions so there was unexpected API changes. The recent insane
> changes also stopped handling DEBIAN files correctly leading to
> races.
> This is perhaps the best backtrace we have:
> 
> https://valkyrie.yoctoproject.org//#/builders/43/builds/246
> 
> Who/Plan: RP has a partial revert for the first part to fix builds.
> Ross is aware and working on the second.

I tested Ross' patch for this and unfortunate it caused a new problem
to appear.

I accidentally triggered two a-full builds at once, there was also a
scarthgap build running at the time.

The patch from Ross caused all the do_package_qa tasks to rerun at once
in parallel which caused writes to the hash equivalence server. This
caused widespread timeout errors and took out all the builds including
the scarthgap one in parallel.

https://valkyrie.yoctoproject.org/#/builders/4/builds/257
https://valkyrie.yoctoproject.org/#/builders/2/builds/253
https://valkyrie.yoctoproject.org/#/builders/89/builds/236/steps/16/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/40/builds/242/steps/11/logs/stdio

and many more.

Joshua: Is there a performance issue with many writes in parallel to
hashequiv?

The stats info off the server was:

    "requests": {
        "average": 0.042532518429623176,
        "max_time": 3.0534732329833787,
        "num": 73325629,
        "stdev": 0.06959859907598737,
        "total_time": 3118723.6668062117
    }

Cheers,

Richard

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

On Sun, 2024-10-13 at 16:26 +0100, Richard Purdie via
lists.openembedded.org wrote:
> I've spent quite a bit of time trying to reproduce/debug this:
> 
> https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
> 
> 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_apt: FAILED (218.02s)
> 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
> 
> on debian11-vk-1.
> 
> I've a successful test and a failed test testimage output for
> comparison:
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465
> 
> along with qemu serial output:
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106
> 
> I've tried rerunning that exact series of tests on that worker with
> no
> success reproducing the issue. The image was overwritten by a later
> test so we can't retest the exact image. I've checked the journal on
> that system and there isn't anything related around the time this
> happened. The two failures were on different network interfaces and
> both interfaces work on later tests.

This is happening on most builds on debian11 for my test branch. If I
add new changes, the issue doesn't happen so it looks to be timing
related and needs a populated sstate cache.

I put auditing into runqemu's tap locks codepaths and the devices are
being locked/released correctly, there is no duplicate device usage.

I also went through the non-tap/tun codepaths and I can't spot any
issues, we do use slirp in some tests.

Since we have a trigger point where we know it is failing (the runtime
ping and ssh tests), I added a os.system("ps awx") into it. That gave
me a process dump of what was running when this happens.

https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3603268
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3902903
https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.4042146

From those we can conclude that there is only one qemu-system-* running
using the appropriate interface. I couldn't spot any other particularly
untoward processes running. That tells us a lot of things it isn't I
guess.

In addition to the ps, I've now added a netstat -tunape to see if we
can see something holding a network connection open. I'm wondering
about our httpserver used in some tests, or debuginfod, or something.
That build is ongoing.

> I'm open to ideas...

I still am, this one is proving to be a pain...

Cheers,

Richard

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

On Sat, 2024-10-19 at 12:05 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Sun, 2024-10-13 at 16:26 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > I've spent quite a bit of time trying to reproduce/debug this:
> > 
> > https://valkyrie.yoctoproject.org//#/builders/35/builds/216/steps/14/logs/stdio
> > 
> > 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> > runtime_test.TestImage.test_testimage_apt: FAILED (218.02s)
> > 2024-10-13 04:12:32,010 - oe-selftest - INFO - RESULTS -
> > runtime_test.TestImage.test_testimage_dnf: FAILED (155.47s)
> > 
> > on debian11-vk-1.
> > 
> > I've a successful test and a failed test testimage output for
> > comparison:
> > 
> > https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.1487387
> > https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/log.do_testimage.3323465
> > 
> > along with qemu serial output:
> > 
> > https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log-fail
> > https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/qemu_boot_log.20241013104106
> > 
> > I've tried rerunning that exact series of tests on that worker with
> > no
> > success reproducing the issue. The image was overwritten by a later
> > test so we can't retest the exact image. I've checked the journal
> > on
> > that system and there isn't anything related around the time this
> > happened. The two failures were on different network interfaces and
> > both interfaces work on later tests.
> 
> This is happening on most builds on debian11 for my test branch. If I
> add new changes, the issue doesn't happen so it looks to be timing
> related and needs a populated sstate cache.
> 
> I put auditing into runqemu's tap locks codepaths and the devices are
> being locked/released correctly, there is no duplicate device usage.
> 
> I also went through the non-tap/tun codepaths and I can't spot any
> issues, we do use slirp in some tests.
> 
> Since we have a trigger point where we know it is failing (the
> runtime
> ping and ssh tests), I added a os.system("ps awx") into it. That gave
> me a process dump of what was running when this happens.
> 
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3603268
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.3902903
> https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/3/log.do_testimage.4042146
> 
> From those we can conclude that there is only one qemu-system-*
> running
> using the appropriate interface. I couldn't spot any other
> particularly
> untoward processes running. That tells us a lot of things it isn't I
> guess.
> 
> In addition to the ps, I've now added a netstat -tunape to see if we
> can see something holding a network connection open. I'm wondering
> about our httpserver used in some tests, or debuginfod, or something.
> That build is ongoing.

https://valkyrie.yocto.io/pub/shared-failure-data/debian11-vk-1-selftest/5/log.do_testimage.123265

This has netstat from the host but also uses the serial connection to
run ping on the host/server IPs and ifconfig.

This tells is the interfaces have the addresses we expect on both sides
and that packets are tx'd but not rx'd on the other side.

Creative ideas on further debugging welcome, I'm not sure where from
here. I think it narrows down the issue to the host kernel or qemu and
these qemu binaries are working on the other builders. The host is a
5.10.223 kernel:

$ cat /proc/version 
Linux version 5.10.0-32-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)

Cheers,

Richard

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Alexander Kanavin]

On Sat, 19 Oct 2024 at 23:48, Richard Purdie via
lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:

> Creative ideas on further debugging welcome, I'm not sure where from
> here. I think it narrows down the issue to the host kernel or qemu and
> these qemu binaries are working on the other builders. The host is a
> 5.10.223 kernel:
>
> $ cat /proc/version
> Linux version 5.10.0-32-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)

There's an option to use 6.1 kernel on bullseye:
https://packages.debian.org/bullseye-backports/linux-image-amd64

so I'd give that a shot, if only to confirm (or refute) it's
definitely 5.10 on the host.

Alex

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

On Mon, 2024-10-21 at 12:55 +0200, Alexander Kanavin wrote:
> On Sat, 19 Oct 2024 at 23:48, Richard Purdie via
> lists.openembedded.org
> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> 
> > Creative ideas on further debugging welcome, I'm not sure where
> > from
> > here. I think it narrows down the issue to the host kernel or qemu
> > and
> > these qemu binaries are working on the other builders. The host is
> > a
> > 5.10.223 kernel:
> > 
> > $ cat /proc/version
> > Linux version 5.10.0-32-amd64 (debian-kernel@lists.debian.org)
> > (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for
> > Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)
> 
> There's an option to use 6.1 kernel on bullseye:
> https://packages.debian.org/bullseye-backports/linux-image-amd64
> 
> so I'd give that a shot, if only to confirm (or refute) it's
> definitely 5.10 on the host.

Michael was able to switch debian11-vk-1 to the 6.1 kernel (thanks!)
and I've run  5 builds successfully on it, the single build I tried on
debian11-vk3 at the same time failed.

I'm therefore willing to say this is some bug in the 5.10 kernel,
switch our other two workers to 6.1 and call things "resolved".

Yes, ideally we'd track it down and get it fixed properly but we're not
really setup to debug Debian bugs in old releases and we have other
more pressing problems.

Cheers,

Richard

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Michael Halstead]

[-- Attachment #1: Type: text/plain, Size: 1775 bytes --]

On Tue, Oct 22, 2024 at 2:58 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2024-10-21 at 12:55 +0200, Alexander Kanavin wrote:
> > On Sat, 19 Oct 2024 at 23:48, Richard Purdie via
> > lists.openembedded.org
> > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> >
> > > Creative ideas on further debugging welcome, I'm not sure where
> > > from
> > > here. I think it narrows down the issue to the host kernel or qemu
> > > and
> > > these qemu binaries are working on the other builders. The host is
> > > a
> > > 5.10.223 kernel:
> > >
> > > $ cat /proc/version
> > > Linux version 5.10.0-32-amd64 (debian-kernel@lists.debian.org)
> > > (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for
> > > Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)
> >
> > There's an option to use 6.1 kernel on bullseye:
> > https://packages.debian.org/bullseye-backports/linux-image-amd64
> >
> > so I'd give that a shot, if only to confirm (or refute) it's
> > definitely 5.10 on the host.
>
> Michael was able to switch debian11-vk-1 to the 6.1 kernel (thanks!)
> and I've run  5 builds successfully on it, the single build I tried on
> debian11-vk3 at the same time failed.
>
> I'm therefore willing to say this is some bug in the 5.10 kernel,
> switch our other two workers to 6.1 and call things "resolved".
>
>
All three Debian 11 Bullseye workers are set up with 6.1.0 kernels now.


> Yes, ideally we'd track it down and get it fixed properly but we're not
> really setup to debug Debian bugs in old releases and we have other
> more pressing problems.
>
> Cheers,
>
> Richard
>


-- 
Michael Halstead
Linux Foundation / Yocto Project
Systems Operations Engineer

[-- Attachment #2: Type: text/html, Size: 2981 bytes --]

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2231 bytes --]

On Tue, Oct 22, 2024 at 2:58 AM Richard Purdie via lists.openembedded.org
<richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:

> On Mon, 2024-10-21 at 12:55 +0200, Alexander Kanavin wrote:
> > On Sat, 19 Oct 2024 at 23:48, Richard Purdie via
> > lists.openembedded.org
> > <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> >
> > > Creative ideas on further debugging welcome, I'm not sure where
> > > from
> > > here. I think it narrows down the issue to the host kernel or qemu
> > > and
> > > these qemu binaries are working on the other builders. The host is
> > > a
> > > 5.10.223 kernel:
> > >
> > > $ cat /proc/version
> > > Linux version 5.10.0-32-amd64 (debian-kernel@lists.debian.org)
> > > (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for
> > > Debian) 2.35.2) #1 SMP Debian 5.10.223-1 (2024-08-10)
> >
> > There's an option to use 6.1 kernel on bullseye:
> > https://packages.debian.org/bullseye-backports/linux-image-amd64
> >
> > so I'd give that a shot, if only to confirm (or refute) it's
> > definitely 5.10 on the host.
>
> Michael was able to switch debian11-vk-1 to the 6.1 kernel (thanks!)
> and I've run  5 builds successfully on it, the single build I tried on
> debian11-vk3 at the same time failed.
>
> I'm therefore willing to say this is some bug in the 5.10 kernel,
> switch our other two workers to 6.1 and call things "resolved".
>
> Yes, ideally we'd track it down and get it fixed properly but we're not
> really setup to debug Debian bugs in old releases and we have other
> more pressing problems.


I agree with this. I was wondering if there is some aspects that Debian
community can be made aware of the problem ?

>
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#206146):
> https://lists.openembedded.org/g/openembedded-core/message/206146
> Mute This Topic: https://lists.openembedded.org/mt/108984574/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3767 bytes --]

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Alexander Kanavin]

On Wed, 23 Oct 2024 at 17:29, Khem Raj <raj.khem@gmail.com> wrote:

>> Yes, ideally we'd track it down and get it fixed properly but we're not
>> really setup to debug Debian bugs in old releases and we have other
>> more pressing problems.
>
>
> I agree with this. I was wondering if there is some aspects that Debian community can be made aware of the problem ?

If we can't make it easy for them to reproduce (and we can't), then
reporting is not likely to achieve much. WONTFIX WORKSFORME and all
that.

Alex

Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie]

On Wed, 2024-10-23 at 17:33 +0200, Alexander Kanavin wrote:
> On Wed, 23 Oct 2024 at 17:29, Khem Raj <raj.khem@gmail.com> wrote:
> 
> > > Yes, ideally we'd track it down and get it fixed properly but
> > > we're not
> > > really setup to debug Debian bugs in old releases and we have
> > > other
> > > more pressing problems.
> > 
> > 
> > I agree with this. I was wondering if there is some aspects that
> > Debian community can be made aware of the problem ?
> 
> If we can't make it easy for them to reproduce (and we can't), then
> reporting is not likely to achieve much. WONTFIX WORKSFORME and all
> that.

It is also in Debian 11 (old stable), not Debian 12 so it is getting
fairly old now. I suspect there will be less interest in the older
release.

Cheers,

Richard