Re: [OE-core] [RFC][PATCH] sstatesig.py: show an error instead of warning when sstate manifest isn't found

Re: [OE-core] [RFC][PATCH] sstatesig.py: show an error instead of warning when sstate manifest isn't found

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3252 bytes --]

On Tue, Feb 23, 2021 at 02:35:53PM +0100, Martin Jansa via lists.openembedded.org wrote:
> * not sure if there are some valid use-cases for missing manifest, but
>   recently I'm seeing increasing number of build failures where something
>   from native recipe is missing (seen it with pseudo, autoconf, nodejs
>   recently) and the only indication that something is wrong (before showing
>   sometimes misleading error like:
>   recipe-sysroot-native/usr/bin/node: No such file or directory
>   is this warning:
>   NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
>   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
>   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
>   NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
>   NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
>   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
>   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded
> 
>   if I rebuild that native dependency, then it gets fixed and I don't
>   see these failures in clean builds (as without sstate and with empty
>   TMPDIR), only in incremental builds
> 
> * but if there isn't valid reason for missing manifest file, then I think
>   it would be better to error early (or even bb.fatal())

Looks like this warning was first introduced in 2014:
https://git.openembedded.org/openembedded-core/commit/?id=79ea036de331bde65a88fb777647dc099ef05acf
then extended and updated for RSS in:
https://git.openembedded.org/openembedded-core/commit/?id=809746f56df4b91af014bf6a3f28997d6698ac78
then added for package management in:
https://git.openembedded.org/openembedded-core/commit/?id=85e72e129362db896b0d368077033e4a2e373cf9
and then unified last 2 to common function in:
https://git.openembedded.org/openembedded-core/commit/?id=68150bac7444f089f19c789e9f6602d59f605d7a

> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/lib/oe/sstatesig.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index 84999ee94d..6cd6e11acc 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
>          manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
>          if os.path.exists(manifest):
>              return manifest, d2
> -    bb.warn("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
> +    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
>      return None, d2
>
>  def OEOuthashBasic(path, sigfile, task, d):
> -- 
> 2.27.0
> 

> 
> 
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [OE-core] [RFC][PATCH] sstatesig.py: show an error instead of warning when sstate manifest isn't found

[Richard Purdie]

On Tue, 2021-02-23 at 14:51 +0100, Martin Jansa wrote:
> On Tue, Feb 23, 2021 at 02:35:53PM +0100, Martin Jansa via lists.openembedded.org wrote:
> > * not sure if there are some valid use-cases for missing manifest, but
> >   recently I'm seeing increasing number of build failures where something
> >   from native recipe is missing (seen it with pseudo, autoconf, nodejs
> >   recently) and the only indication that something is wrong (before showing
> >   sometimes misleading error like:
> >   recipe-sysroot-native/usr/bin/node: No such file or directory
> >   is this warning:
> >   NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
> >   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
> >   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
> >   NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
> >   NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
> >   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
> >   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded
> > 
> >   if I rebuild that native dependency, then it gets fixed and I don't
> >   see these failures in clean builds (as without sstate and with empty
> >   TMPDIR), only in incremental builds
> > 
> > * but if there isn't valid reason for missing manifest file, then I think
> >   it would be better to error early (or even bb.fatal())
> 
> Looks like this warning was first introduced in 2014:
> https://git.openembedded.org/openembedded-core/commit/?id=79ea036de331bde65a88fb777647dc099ef05acf
> then extended and updated for RSS in:
> https://git.openembedded.org/openembedded-core/commit/?id=809746f56df4b91af014bf6a3f28997d6698ac78
> then added for package management in:
> https://git.openembedded.org/openembedded-core/commit/?id=85e72e129362db896b0d368077033e4a2e373cf9
> and then unified last 2 to common function in:
> https://git.openembedded.org/openembedded-core/commit/?id=68150bac7444f089f19c789e9f6602d59f605d7a

I merged this. There were cases where the warning showed in cases where
there wasn't a problem but I think we've fixed all those now.

Cheers,

Richard

Re: [OE-core] [RFC][PATCH] sstatesig.py: show an error instead of warning when sstate manifest isn't found

[Denys Dmytriyenko]

On Fri, Feb 26, 2021 at 03:53:38PM +0000, Richard Purdie wrote:
> On Tue, 2021-02-23 at 14:51 +0100, Martin Jansa wrote:
> > On Tue, Feb 23, 2021 at 02:35:53PM +0100, Martin Jansa via lists.openembedded.org wrote:
> > > * not sure if there are some valid use-cases for missing manifest, but
> > >   recently I'm seeing increasing number of build failures where something
> > >   from native recipe is missing (seen it with pseudo, autoconf, nodejs
> > >   recently) and the only indication that something is wrong (before showing
> > >   sometimes misleading error like:
> > >   recipe-sysroot-native/usr/bin/node: No such file or directory
> > >   is this warning:
> > >   NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
> > >   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
> > >   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
> > >   NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
> > >   NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
> > >   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
> > >   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded
> > > 
> > >   if I rebuild that native dependency, then it gets fixed and I don't
> > >   see these failures in clean builds (as without sstate and with empty
> > >   TMPDIR), only in incremental builds
> > > 
> > > * but if there isn't valid reason for missing manifest file, then I think
> > >   it would be better to error early (or even bb.fatal())
> > 
> > Looks like this warning was first introduced in 2014:
> > https://git.openembedded.org/openembedded-core/commit/?id=79ea036de331bde65a88fb777647dc099ef05acf
> > then extended and updated for RSS in:
> > https://git.openembedded.org/openembedded-core/commit/?id=809746f56df4b91af014bf6a3f28997d6698ac78
> > then added for package management in:
> > https://git.openembedded.org/openembedded-core/commit/?id=85e72e129362db896b0d368077033e4a2e373cf9
> > and then unified last 2 to common function in:
> > https://git.openembedded.org/openembedded-core/commit/?id=68150bac7444f089f19c789e9f6602d59f605d7a
> 
> I merged this. There were cases where the warning showed in cases where
> there wasn't a problem but I think we've fixed all those now.

Richard, Martin,

Can you please point me to the right way of fixing this? I've been seeing 
this warning before and since it is now an error and backported to Dunfell, 
it breaks builds one in a while.

-- 
Denys

Re: [OE-core] [RFC][PATCH] sstatesig.py: show an error instead of warning when sstate manifest isn't found

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3326 bytes --]

On Fri, Mar 12, 2021 at 05:41:50PM -0500, Denys Dmytriyenko wrote:
> On Fri, Feb 26, 2021 at 03:53:38PM +0000, Richard Purdie wrote:
> > On Tue, 2021-02-23 at 14:51 +0100, Martin Jansa wrote:
> > > On Tue, Feb 23, 2021 at 02:35:53PM +0100, Martin Jansa via lists.openembedded.org wrote:
> > > > * not sure if there are some valid use-cases for missing manifest, but
> > > >   recently I'm seeing increasing number of build failures where something
> > > >   from native recipe is missing (seen it with pseudo, autoconf, nodejs
> > > >   recently) and the only indication that something is wrong (before showing
> > > >   sometimes misleading error like:
> > > >   recipe-sysroot-native/usr/bin/node: No such file or directory
> > > >   is this warning:
> > > >   NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
> > > >   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
> > > >   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
> > > >   NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
> > > >   NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
> > > >   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
> > > >   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded
> > > > 
> > > >   if I rebuild that native dependency, then it gets fixed and I don't
> > > >   see these failures in clean builds (as without sstate and with empty
> > > >   TMPDIR), only in incremental builds
> > > > 
> > > > * but if there isn't valid reason for missing manifest file, then I think
> > > >   it would be better to error early (or even bb.fatal())
> > > 
> > > Looks like this warning was first introduced in 2014:
> > > https://git.openembedded.org/openembedded-core/commit/?id=79ea036de331bde65a88fb777647dc099ef05acf
> > > then extended and updated for RSS in:
> > > https://git.openembedded.org/openembedded-core/commit/?id=809746f56df4b91af014bf6a3f28997d6698ac78
> > > then added for package management in:
> > > https://git.openembedded.org/openembedded-core/commit/?id=85e72e129362db896b0d368077033e4a2e373cf9
> > > and then unified last 2 to common function in:
> > > https://git.openembedded.org/openembedded-core/commit/?id=68150bac7444f089f19c789e9f6602d59f605d7a
> > 
> > I merged this. There were cases where the warning showed in cases where
> > there wasn't a problem but I think we've fixed all those now.
> 
> Richard, Martin,
> 
> Can you please point me to the right way of fixing this? I've been seeing 
> this warning before and since it is now an error and backported to Dunfell, 
> it breaks builds one in a while.

Do you see this error triggered on initramfs manifest used by your
kernel? See this thread for possible work arounds:
https://lists.openembedded.org/g/openembedded-core/topic/81249042#149271

but maybe you have some different corner case which will need different
fix.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [OE-core] [RFC][PATCH] sstatesig.py: show an error instead of warning when sstate manifest isn't found

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 7659 bytes --]

On Tue, Feb 23, 2021 at 02:35:53PM +0100, Martin Jansa via lists.openembedded.org wrote:
> * not sure if there are some valid use-cases for missing manifest, but
>   recently I'm seeing increasing number of build failures where something
>   from native recipe is missing (seen it with pseudo, autoconf, nodejs
>   recently) and the only indication that something is wrong (before showing
>   sometimes misleading error like:
>   recipe-sysroot-native/usr/bin/node: No such file or directory
>   is this warning:
>   NOTE: Running task 7844 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_prepare_recipe_sysroot)
>   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Started
>   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
>   NOTE: Running task 7845 of 12431 (/meta-oe/meta-oe/recipes-devtools/nodejs/nodejs_12.20.2.bb:do_unpack)
>   NOTE: recipe nodejs-12.20.2-r0: task do_unpack: Started
>   WARNING: nodejs-12.20.2-r0 do_prepare_recipe_sysroot: Manifest /BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-nodejs-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
>   NOTE: recipe nodejs-12.20.2-r0: task do_prepare_recipe_sysroot: Succeeded
> 
>   if I rebuild that native dependency, then it gets fixed and I don't
>   see these failures in clean builds (as without sstate and with empty
>   TMPDIR), only in incremental builds
> 
> * but if there isn't valid reason for missing manifest file, then I think
>   it would be better to error early (or even bb.fatal())

FYI:

With bb.fatal() I'm using for my builds I got few more failures now
(which might have been silently ignored like nodejs-native in example
before), this time it was
image.do_prepare_recipe_sysroot:
  ERROR: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-bmap-tools-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?

linux-yocto.do_validate_branches
  ERROR: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-kern-tools-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
u-boot.do_prepare_recipe_sysroot
  ERROR: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-kern-tools-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?
busybox.do_prepare_recipe_sysroot
  NOTE: Direct dependencies are ['TOPDIR/oe-core/meta/recipes-devtools/gcc/gcc-cross_10.2.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-devtools/gcc/gcc-runtime_10.2.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.3.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-core/glibc/glibc_2.33.bb:do_populate_sysroot', 'virtual:native:TOPDIR/oe-core/meta/recipes-devtools/icecc-create-env/icecc-create-env_0.1.bb:do_populate_sysroot', 'virtual:native:TOPDIR/oe-core/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'TOPDIR/oe-core/meta/recipes-core/libxcrypt/libxcrypt_4.4.18.bb:do_populate_sysroot']
  ERROR: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_ubuntu-18.04-kern-tools-native.populate_sysroot not found in x86_64 x86_64_ubuntu-18.04 (variant '')?

gstreamer1.0-plugins-base.do_prepare_recipe_sysroot
  ERROR: gstreamer1.0-plugins-base-1.18.3-r0 do_prepare_recipe_sysroot: Manifest TOPDIR/BUILD/sstate-control/manifest-x86_64_x86_64-nativesdk-iso-codes.populate_sysroot

And the issue with kern-tools-native was reproduced in 3 different workspaces on different builders
which don't share the same SSTATE_MIRROR.

I'll try to debug what happened in these cases, but it was again on jenkins where the workspace
was already overwritten by some other builds and the evidence might be lost already and last time
I've tried to reproduce this locally it never failed.

I'm pretty sure that previous successful build which left the TMPDIR as well as SSTATE_DIR populated was
based on oe-core b27bff26a9 (+ some local modifications) and the current failing once is
69f8f3e213 (+ similar local modifications), and the git log shows that bmap-tools, kern-tools
and iso-codes were updated in this:

$ git log --oneline b27bff26a9..69f8f3e213 | tee
69f8f3e213 bmap-tools: upgrade 3.5 -> 3.6
09cfa5ce1e spirv-tools: Replace strncpy with memcpy
f939624981 local.conf.sample.extended: sample value for SDK_PS1
81ed52f245 toolchain-scripts.bbclass: customize prompt string for SDKs
1d19b16846 sanity.bbclass: Add az fetcher to accepted URI protocols for mirrors sanity
7639ffed1a iso-codes: upgrade 4.5.0 -> 4.6.0
ad4e023bb3 help2man: upgrade 1.48.1 -> 1.48.2
a7d7c1412b sysklogd: upgrade 2.2.1 -> 2.2.2
d34cc88cb7 python3-zipp: upgrade 3.4.0 -> 3.4.1
27da7a57b0 python3-pygments: upgrade 2.8.0 -> 2.8.1
9f8d049dbb runqemu: modify novga to conform to documentation
9dea4cd2f9 runqemu: correct forcing of ttyS0
695c98b652 runqemu: add QB_GRAPHICS
7290b77348 meta/recipes-devtools: Add HOMEPAGE / DESCRIPTION
5ab087950c meta/recipes-graphics: Add HOMEPAGE / DESCRIPTION
ab7a945f6e meta/recipes-multimedia: Add HOMEPAGE / DESCRIPTION
693988ec42 meta/recipes-kernel: Add HOMEPAGE / DESCRIPTION
325db7fea1 meta/recipes-support: Add HOMEPAGE / DESCRIPTION
ffdce193f3 ffmpeg: upgrade 4.3.1 -> 4.3.2
07c09631d8 linux-yocto/qemuarmv5: fix configuration warning
7a48aad3d3 linux-yocto/5.10: update to v5.10.21
eea536b436 kern-tools: symbol-why fix and README update
820b552341 linux-yocto/5.4: update to v5.4.103
f9a5dc7eeb parselogs: Allow expected kernel messages for qemuppc64
be3a6b9791 inittab: Add getty launch on hvc0 for qemuppc64
13bc8fa9d7 formfactor: Add machine config for qemuppc64
3a741b0f39 linux-yocto: Enable powerpc64le QEMU
68275b25f0 qemuppc64: Add a QEMU machine definition for ppc64
20d9e13372 qemu: fix CVE-2021-20203
5610a5b4ef machine: Initial commit of qemuriscv32
8a40989f03 linux-yocto: Add qemuriscv32 as a compatible machine
22691df60a openssl: Enable building for RISC-V 32-bit
74e5c43802 buildstats.bbclass: improve timeout handling

At least I'm seeing this only with master, I'm using the same bb.fatal in dunfell and gatesgarth
based builds and so far those triggered only the issue with initramfs discussed in the other thread.

It might be just being more lucky as there are fewer changes in dunfell/gatesgarth compared with
master.

Cheers,

> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/lib/oe/sstatesig.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index 84999ee94d..6cd6e11acc 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2, taskname, d, multilibcache):
>          manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" % (pkgarch, taskdata, taskname))
>          if os.path.exists(manifest):
>              return manifest, d2
> -    bb.warn("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
> +    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest, d2.expand(" ".join(pkgarchs)), variant))
>      return None, d2
>
>  def OEOuthashBasic(path, sigfile, task, d):
> -- 
> 2.27.0
> 

> 
> 
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 201 bytes --]