[PATCH 0/7] Remove 'extend_recipe_sysroot' from 'BB_HASHEXCLUDE_COMMON'

[PATCH 0/7] Remove 'extend_recipe_sysroot' from 'BB_HASHEXCLUDE_COMMON'

[Adam Blank]

It has lain there for a while, obscuring certain aspects of
how task signatures are calculated.

This changeset does not change the current status quo, but
rather changes the way in which it is implemented. It can be
followed by changes which further refine what is excluded and
how, from those tasks which refer to 'extend_recipe_sysroot'.

Related discussion:
https://lore.kernel.org/openembedded-core/aa893eb861bf4dd18e3e84e6bbebdee3b0367d1b.camel@linuxfoundation.org/

Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
---
Adam Blank (7):
      package_pkgdata: fix typo to stop calling undefined function
      staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
      staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
      cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
      native: add 'extend_recipe_sysroot' to 'vardepsexclude'
      wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude'
      bitbake.conf: remove 'extend_recipe_sysroot' from BB_HASHEXCLUDE_COMMON

 meta/classes-global/package_pkgdata.bbclass | 4 +++-
 meta/classes-global/staging.bbclass         | 5 ++++-
 meta/classes-recipe/cross.bbclass           | 1 +
 meta/classes-recipe/native.bbclass          | 1 +
 meta/conf/bitbake.conf                      | 2 +-
 meta/recipes-core/meta/wic-tools.bb         | 1 +
 6 files changed, 11 insertions(+), 3 deletions(-)
---
base-commit: d9b69d0a2d69e003b9432e1473830a89ff5a06c4
change-id: 20260418-extend_recipe_sysroot-d958ba9cfa9a

Best regards,
-- 
Adam Blank <adam.blank.g@gmail.com>

[PATCH 1/7] package_pkgdata: fix typo to stop calling undefined function

[Adam Blank]

The function is named 'package_populate_pkgdata_dir' but the
call was to 'staging_package_populate_pkgdata_dir'.

Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
---
 meta/classes-global/package_pkgdata.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes-global/package_pkgdata.bbclass b/meta/classes-global/package_pkgdata.bbclass
index f653bd9240..b2f4e7594c 100644
--- a/meta/classes-global/package_pkgdata.bbclass
+++ b/meta/classes-global/package_pkgdata.bbclass
@@ -31,6 +31,8 @@ def package_populate_pkgdata_dir(pkgdatadir, d):
                     except FileExistsError:
                         continue
 
+package_populate_pkgdata_dir[vardepsexclude] += "MACHINE_ARCH PACKAGE_EXTRA_ARCHS"
+
 python package_prepare_pkgdata() {
     import copy
     import glob
@@ -47,7 +49,7 @@ python package_prepare_pkgdata() {
     # Detect bitbake -b usage
     nodeps = d.getVar("BB_LIMITEDDEPS") or False
     if nodeps:
-        staging_package_populate_pkgdata_dir(pkgdatadir, d)
+        package_populate_pkgdata_dir(pkgdatadir, d)
         return
 
     start = None

-- 
2.43.0

[PATCH 2/7] staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'

[Adam Blank]

It will be needed when 'extend_recipe_sysroot' is no longer
ignored when calculating signatures.

Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
---
 meta/classes-global/staging.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes-global/staging.bbclass b/meta/classes-global/staging.bbclass
index 1008867a6c..259603dcbf 100644
--- a/meta/classes-global/staging.bbclass
+++ b/meta/classes-global/staging.bbclass
@@ -248,6 +248,8 @@ def staging_populate_sysroot_dir(targetsysroot, nativesysroot, native, d):
     for p in sorted(postinsts):
         bb.note("Running postinst {}, output:\n{}".format(p, subprocess.check_output(p, shell=True, stderr=subprocess.STDOUT)))
 
+staging_populate_sysroot_dir[vardepsexclude] += "PACKAGE_EXTRA_ARCHS"
+
 #
 # Manifests here are complicated. The main sysroot area has the unpacked sstate
 # which us unrelocated and tracked by the main sstate manifests. Each recipe

-- 
2.43.0

[PATCH 3/7] staging: add 'extend_recipe_sysroot' to 'vardepsexclude'

[Adam Blank]

'do_prepare_recipe_sysroot' did not declare this explicitly,
but made use of the fact, that 'extend_recipe_sysroot' is
ignored on the base hash level.

Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
---
 meta/classes-global/staging.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes-global/staging.bbclass b/meta/classes-global/staging.bbclass
index 259603dcbf..3684e76bd7 100644
--- a/meta/classes-global/staging.bbclass
+++ b/meta/classes-global/staging.bbclass
@@ -645,12 +645,13 @@ python extend_recipe_sysroot() {
 
     bb.utils.unlockfile(lock)
 }
-extend_recipe_sysroot[vardepsexclude] += "MACHINE_ARCH PACKAGE_EXTRA_ARCHS SDK_ARCH BUILD_ARCH SDK_OS BB_TASKDEPDATA"
+extend_recipe_sysroot[vardepsexclude] += "BB_TASKDEPDATA"
 
 do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"
 python do_prepare_recipe_sysroot () {
     bb.build.exec_func("extend_recipe_sysroot", d)
 }
+do_prepare_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot"
 addtask do_prepare_recipe_sysroot before do_configure after do_fetch
 
 python staging_taskhandler() {

-- 
2.43.0

[PATCH 4/7] cross: add 'extend_recipe_sysroot' to 'vardepsexclude'

[Adam Blank]

'do_addto_recipe_sysroot' did not declare this explicitly,
but made use of the fact, that 'extend_recipe_sysroot' is
ignored on the base hash level.

Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
---
 meta/classes-recipe/cross.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes-recipe/cross.bbclass b/meta/classes-recipe/cross.bbclass
index 82a49dc8f5..bceee5d6e6 100644
--- a/meta/classes-recipe/cross.bbclass
+++ b/meta/classes-recipe/cross.bbclass
@@ -98,6 +98,7 @@ python do_addto_recipe_sysroot () {
 }
 addtask addto_recipe_sysroot after do_populate_sysroot
 do_addto_recipe_sysroot[deptask] = "do_populate_sysroot"
+do_addto_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot"
 
 PATH:prepend = "${COREBASE}/scripts/cross-intercept:"
 

-- 
2.43.0

[PATCH 5/7] native: add 'extend_recipe_sysroot' to 'vardepsexclude'

[Adam Blank]

'do_addto_recipe_sysroot' did not declare this explicitly,
but made use of the fact, that 'extend_recipe_sysroot' is
ignored on the base hash level.

Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
---
 meta/classes-recipe/native.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes-recipe/native.bbclass b/meta/classes-recipe/native.bbclass
index 9f4ca170f7..ac9458a2f4 100644
--- a/meta/classes-recipe/native.bbclass
+++ b/meta/classes-recipe/native.bbclass
@@ -176,6 +176,7 @@ python do_addto_recipe_sysroot () {
 }
 addtask addto_recipe_sysroot after do_populate_sysroot
 do_addto_recipe_sysroot[deptask] = "do_populate_sysroot"
+do_addto_recipe_sysroot[vardepsexclude] += "extend_recipe_sysroot"
 
 inherit nopackages
 

-- 
2.43.0

[PATCH 6/7] wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude'

[Adam Blank]

'do_build_sysroot' did not declare this explicitly,
 but made use of the fact, that 'extend_recipe_sysroot' is
ignored on the base hash level.

Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
---
 meta/recipes-core/meta/wic-tools.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/meta/wic-tools.bb b/meta/recipes-core/meta/wic-tools.bb
index 45fb873dd6..1f86347739 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -27,3 +27,4 @@ python do_build_sysroot () {
     bb.build.exec_func("extend_recipe_sysroot", d)
 }
 addtask do_build_sysroot after do_prepare_recipe_sysroot before do_build
+do_build_sysroot[vardepsexclude] += "extend_recipe_sysroot"
\ No newline at end of file

-- 
2.43.0

[PATCH 7/7] bitbake.conf: remove 'extend_recipe_sysroot' from BB_HASHEXCLUDE_COMMON

[Adam Blank]

'extend_recipe_sysroot' can and should be handled just like any other
function. Fine grained use of 'vardepsexclude' is capable of achieving
the same effect as ingoring 'extend_recipe_sysroot' on the base hash
level, but with a better visibility, and is a more uniform approach.

Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9fcd16615d..87c6a700d2 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -961,7 +961,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     STAMPS_DIR PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
     CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
     WARN_QA WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \
-    BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
+    BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH DEPLOY_DIR \
     SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
     SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
     OMP_NUM_THREADS BB_CURRENTTASK"

-- 
2.43.0

Re: [OE-core] [PATCH 0/7] Remove 'extend_recipe_sysroot' from 'BB_HASHEXCLUDE_COMMON'

[Richard Purdie]

On Sat, 2026-04-18 at 20:34 +0200, Adam Blank via lists.openembedded.org wrote:
> It has lain there for a while, obscuring certain aspects of
> how task signatures are calculated.
> 
> This changeset does not change the current status quo, but
> rather changes the way in which it is implemented. It can be
> followed by changes which further refine what is excluded and
> how, from those tasks which refer to 'extend_recipe_sysroot'.
> 
> Related discussion:
> https://lore.kernel.org/openembedded-core/aa893eb861bf4dd18e3e84e6bbebdee3b0367d1b.camel@linuxfoundation.org/
> 
> Signed-off-by: Adam Blank <adam.blank.g@gmail.com>
> ---
> Adam Blank (7):
>       package_pkgdata: fix typo to stop calling undefined function
>       staging: add 'vardepsexclude' to 'staging_populate_sysroot_dir'
>       staging: add 'extend_recipe_sysroot' to 'vardepsexclude'
>       cross: add 'extend_recipe_sysroot' to 'vardepsexclude'
>       native: add 'extend_recipe_sysroot' to 'vardepsexclude'
>       wic-tool: add 'extend_recipe_sysroot' to 'vardepsexclude'
>       bitbake.conf: remove 'extend_recipe_sysroot' from
> BB_HASHEXCLUDE_COMMON

This change does have much wider impact than it would first appear.
Even just looking at one of the first tasks in a build, quilt-
native:do_fetch, generating sig data before and after these patches
shows:

bitbake-diffsigs tmp/stamps/x86_64-linux/quilt-native/0.69.do_fetch.sigdata.*
NOTE: Starting bitbake server...
basehash_ignore_vars changed from 'frozenset({'LOGNAME', 'BUILD_ARCH', 'PARALLEL_MAKE', 'USER', 'PRSERV_LOCKDOWN', 'BB_WORKERCONTEXT', 'RUST_BUILD_SYS', 'THISDIR', 'COREBASE', 'EXTERNAL_TOOLCHAIN', 'WARN_QA', 'FILESPATH', 'CCACHE_NOHASHDIR', 'BBPATH', 'SHELL', 'STAMPCLEAN', 'CCACHE', 'DEPLOY_DIR', 'LICENSE_PATH', 'OMP_NUM_THREADS', 'SSTATE_PKGARCH', 'STAGING_DIR_HOST', 'PRSERV_HOST', 'BUILDHISTORY_DIR', 'SOURCE_DATE_EPOCH', 'BBSERVER', 'BB_LIMITEDDEPS', 'HOME', 'BB_HASHSERVE', 'BB_TASKHASH', 'PATH', 'STAGING_DIR_TARGET', 'STAMPS_DIR', 'GIT_CEILING_DIRECTORIES', 'PRSERV_DUMPFILE', 'extend_recipe_sysroot', 'RUST_HOST_SYS', 'TMPDIR', 'CCACHE_DIR', 'SSTATE_HASHEQUIV_OWNER', 'DL_DIR', 'FILESEXTRAPATHS', 'SSTATE_DIR', 'FILE', 'PWD', 'SSTATE_HASHEQUIV_METHOD', 'FILE_DIRNAME', 'PSEUDO_INCLUDE_PATHS', 'WORKDIR', 'CCACHE_TOP_DIR', 'PRSERV_DUMPDIR', 'SDKPKGSUFFIX', 'SSTATE_HASHEQUIV_REPORT_TASKDATA', 'BB_CURRENTTASK', 'PKGDATA_DIR', 'RUST_TARGET_SYS', 'BB_UNIHASH'})' to 'frozenset({'LOGNAME', 'BUILD_ARCH', 'PARALLEL_MAKE', 'USER', 'PRSERV_LOCKDOWN', 'BB_WORKERCONTEXT', 'RUST_BUILD_SYS', 'THISDIR', 'COREBASE', 'EXTERNAL_TOOLCHAIN', 'WARN_QA', 'FILESPATH', 'CCACHE_NOHASHDIR', 'BBPATH', 'SHELL', 'STAMPCLEAN', 'CCACHE', 'DEPLOY_DIR', 'LICENSE_PATH', 'OMP_NUM_THREADS', 'SSTATE_PKGARCH', 'STAGING_DIR_HOST', 'PRSERV_HOST', 'BUILDHISTORY_DIR', 'SOURCE_DATE_EPOCH', 'BBSERVER', 'BB_LIMITEDDEPS', 'HOME', 'BB_HASHSERVE', 'BB_TASKHASH', 'PATH', 'STAGING_DIR_TARGET', 'STAMPS_DIR', 'GIT_CEILING_DIRECTORIES', 'PRSERV_DUMPFILE', 'RUST_HOST_SYS', 'TMPDIR', 'CCACHE_DIR', 'SSTATE_HASHEQUIV_OWNER', 'DL_DIR', 'FILESEXTRAPATHS', 'SSTATE_DIR', 'FILE', 'PWD', 'SSTATE_HASHEQUIV_METHOD', 'FILE_DIRNAME', 'PSEUDO_INCLUDE_PATHS', 'WORKDIR', 'CCACHE_TOP_DIR', 'PRSERV_DUMPDIR', 'SDKPKGSUFFIX', 'SSTATE_HASHEQUIV_REPORT_TASKDATA', 'BB_CURRENTTASK', 'PKGDATA_DIR', 'RUST_TARGET_SYS', 'BB_UNIHASH'})'
changed items: frozenset({'extend_recipe_sysroot'})
Task dependencies changed from:
['OE_SHARED_UMASK', 'PV', 'SRC_URI', 'SRC_URI[sha256sum]', 'base_do_fetch', 'do_fetch[network]', 'do_fetch[umask]', 'fetcher_hashes_dummyfunc']
to:
['BB_RUNTASK', 'COMPONENTS_DIR', 'OE_SHARED_UMASK', 'PN', 'PV', 'RECIPE_SYSROOT', 'RECIPE_SYSROOT_MANIFEST_SUBDIR', 'RECIPE_SYSROOT_NATIVE', 'SRC_URI', 'SRC_URI[sha256sum]', 'SSTATETASKS', 'STAGING_DIR', 'base_do_fetch', 'do_fetch[network]', 'do_fetch[umask]', 'extend_recipe_sysroot', 'fetcher_hashes_dummyfunc', 'oe.path.remove', 'oe.sstatesig.find_sstate_manifest', 'oe.utils.get_multilib_datastore', 'setscene_depvalid', 'sstate_clean_manifest', 'staging_copydir', 'staging_copyfile', 'staging_populate_sysroot_dir', 'staging_processfixme']
basehash changed from fa1058e1378b6cf06cae11a9e251b7ada2d7f81a0de75644c1f693ac842576c0 to a11321c2775abee7697344dca7a79211c5de79a5de60f1e767ed9556ab434982
Dependency on variable BB_RUNTASK was added
Dependency on variable COMPONENTS_DIR was added
Dependency on variable PN was added
Dependency on variable RECIPE_SYSROOT was added
Dependency on variable RECIPE_SYSROOT_MANIFEST_SUBDIR was added
Dependency on variable RECIPE_SYSROOT_NATIVE was added
Dependency on variable SSTATETASKS was added
Dependency on variable STAGING_DIR was added
Dependency on variable extend_recipe_sysroot was added
Dependency on variable oe.path.remove was added
Dependency on variable oe.sstatesig.find_sstate_manifest was added
Dependency on variable oe.utils.get_multilib_datastore was added
Dependency on variable setscene_depvalid was added
Dependency on variable sstate_clean_manifest was added
Dependency on variable staging_copydir was added
Dependency on variable staging_copyfile was added
Dependency on variable staging_populate_sysroot_dir was added
Dependency on variable staging_processfixme was added

which is a lot of non-trivial change and a lot of extra data in the
sigdata file.

Do we really want all of this data to be added into every signature
computation?

Are we 100% sure that add the newly added variable dependencies are
"safe" and won't trigger sstate cache reuse issues?

Right now, I'm very nervous about this.

Cheers,

Richard

Re: [OE-core] [PATCH 0/7] Remove 'extend_recipe_sysroot' from 'BB_HASHEXCLUDE_COMMON'

[Adam Blank]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

>
>
> which is a lot of non-trivial change and a lot of extra data in the
> sigdata file.
>
> Do we really want all of this data to be added into every signature
> computation?
>
> Are we 100% sure that add the newly added variable dependencies are
> "safe" and won't trigger sstate cache reuse issues?
>

Well, obviously such increase in the visibility of dependencies is not a
comforting or welcome thing, but let's not forget, that those dependencies
truly are there :-)
I'd ask the opposite question: given the extent and nature of the relations
exposed by this change, do we want to keep wholesale hiding them with the
current implementation?
Or in other words, are those particular dependencies (stemming from
'extend_recipe_sysroot') so irrelevant for the overall signature and sstate
management, that it is justifiable to keep ignoring them in such a unique
way and on such a fundamental level?

Unless it is truly a desirable situation and the whole subject can be
dismissed, what would be the recommended way to tackle it?


>
> Right now, I'm very nervous about this.
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 1867 bytes --]

Re: [OE-core] [PATCH 0/7] Remove 'extend_recipe_sysroot' from 'BB_HASHEXCLUDE_COMMON'

[Richard Purdie]

On Thu, 2026-04-30 at 17:15 +0200, Adam Blank via
lists.openembedded.org wrote:
> > 
> > which is a lot of non-trivial change and a lot of extra data in the
> > sigdata file.
> > 
> > Do we really want all of this data to be added into every signature
> > computation?
> > 
> > Are we 100% sure that add the newly added variable dependencies are
> > "safe" and won't trigger sstate cache reuse issues?
> > 
> 
> 
> Well, obviously such increase in the visibility of dependencies is
> not a comforting or welcome thing, but let's not forget, that those
> dependencies truly are there :-)
> I'd ask the opposite question: given the extent and nature of the
> relations exposed by this change, do we want to keep wholesale hiding
> them with the current implementation?
> Or in other words, are those particular dependencies (stemming from
> 'extend_recipe_sysroot') so irrelevant for the overall signature and
> sstate management, that it is justifiable to keep ignoring them in
> such a unique way and on such a fundamental level?
> 
> Unless it is truly a desirable situation and the whole subject can be
> dismissed, what would be the recommended way to tackle it?

I'm mentioning this as I wanted to explain the delay, and why I'm
hesitant about this.

Not every piece of build information is put into the hashes and we
previously made a conscious decision that extend_recipe_sysroot and
anything hidden behind it were out of scope for the hashes. Looking at
at the list of new dependencies, I can see why we did that.

Does having these dependencies help the average user? or does it waste
space in the sigdata files and complicate the dependencies? Right now
I'm torn and I'm not sure which way we should proceed.

I should also mention that in testing, we've had sstate mismatch issues
too, e.g. 
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3747

We don't know which of the patches under test are at fault for that and
the errors are non-specific, just that sstate wasn't matching. It could
be something in this patchset, or it might be something else, we just
can't tell and don't know.

Cheers,

Richard