Re: [OE-core] Latest AB-INT unexplained mystery failure

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Mon, 2024-10-14 at 14:46 +0100, Richard Purdie via
lists.openembedded.org wrote:
> It was suggested I should write down a list of other issues we have
> on
> the autobuilder right now. Some of these are related to the
> transition
> to new infrastructure, some are previously seen issues occurring more
> frequently, some are entirely new. I appreciate these should all have
> bugs. I'm copying various people including those helping with SWAT so
> we can transition those we don't get solved into bugzilla.
> 
> a) insane do_package_qa intermittent failures
> ---------------------------------------------
> 
> Two levels of probems. The cachedpath code returns False instead of
> exceptions so there was unexpected API changes. The recent insane
> changes also stopped handling DEBIAN files correctly leading to
> races.
> This is perhaps the best backtrace we have:
> 
> https://valkyrie.yoctoproject.org//#/builders/43/builds/246
> 
> Who/Plan: RP has a partial revert for the first part to fix builds.
> Ross is aware and working on the second.

I tested Ross' patch for this and unfortunate it caused a new problem
to appear.

I accidentally triggered two a-full builds at once, there was also a
scarthgap build running at the time.

The patch from Ross caused all the do_package_qa tasks to rerun at once
in parallel which caused writes to the hash equivalence server. This
caused widespread timeout errors and took out all the builds including
the scarthgap one in parallel.

https://valkyrie.yoctoproject.org/#/builders/4/builds/257
https://valkyrie.yoctoproject.org/#/builders/2/builds/253
https://valkyrie.yoctoproject.org/#/builders/89/builds/236/steps/16/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/40/builds/242/steps/11/logs/stdio

and many more.

Joshua: Is there a performance issue with many writes in parallel to
hashequiv?

The stats info off the server was:

    "requests": {
        "average": 0.042532518429623176,
        "max_time": 3.0534732329833787,
        "num": 73325629,
        "stdev": 0.06959859907598737,
        "total_time": 3118723.6668062117
    }

Cheers,

Richard