Re: [OE-core] [PATCH 1/3] selftest/sstatetests: add tests for 'bitbake -S printdiff'

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Mon, 2023-11-06 at 12:47 +0000, Richard Purdie via
lists.openembedded.org wrote:
> On Thu, 2023-10-26 at 12:33 +0200, Alexander Kanavin wrote:
> > > 'bitbake -S printdiff' is a useful diagnostic facility for finding out
> > > why sstate is not being reused, but until now it had no tests that would
> > > ensure it works. This commit adds three basic scenarios:
> > > 
> > > 1. make a change in a really basic, common recipe that is at the very root
> > > of dependency trees (quilt-native), and ensure that change is correctly discovered when
> > > building an image.
> > > 
> > > 2. make a change in gcc-source recipe, which is somewhat special
> > > (operates in work-shared), and ensure that gcc-runtime builds track
> > > that down as well.
> > > 
> > > 3. make a change in base_do_configure() definition from base.bbclass,
> > > which is not recipe-specific, but affects many basic recipes, and ensure that
> > > is correctly reported as well.
> > > 
> > > The test itself actually runs twice:
> > > - first against a fully populated build directory, where
> > > the printdiff code is guaranteed to find the correct previous
> > > stamp that can be compared with in a predictable manner.
> > > 
> > > - then in an empty build directory where the printdiff code
> > > goes to look in the sstate cache, and so the existence of the
> > > previous signature can be tested, but not the difference with it
> > > (what the exact difference would be is unpredictable as the
> > > sstate cache is indeed shared between many builds).
> > > 
> > > Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> 
> There have been autobulilder failures from the do_configure base change
> and we've not been quite sure why. The summary below took quite a bit
> of digging but seems quite simple when written up like this!
> 
> If you dig into a failure like:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2379/steps/15/logs/stdio
> 
> you see messages like:
> 
> The differences between the current build and any cached tasks start at the following tasks:
> /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_collect_spdx_deps
> virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_configure
> virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/zlib/zlib_1.3.bb:do_collect_spdx_deps
> virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_collect_spdx_deps
> /home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/timezone/tzcode-native.bb:do_prepare_recipe_sysroot
> 
> which seems odd. Taking the first one, if quilt-native has built, you'd
> expect those to be present.
> 
> I went onto the arm builder and tried a "bitbake quilt-native -S printdiff -DDD":
> 
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/4d/8d/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:4d8da10af56c40d61adb5febb96f583a399ceb276d0dc13e61971f1eb0b7d81d_create_runtime_spdx.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/85/79/sstate:quilt-native::0.67:r0::11:85794fd6d83b61cb273e4d14f6e3581d7fa4ec743b455f1d21786cff032c262c_patch.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/b8/fb/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:b8fbbfbafeb7cdf333f1766d6b3c94822a21d42145e17377bede29e44a86a14d_prepare_recipe_sysroot.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fe/ae/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:feae04b2abf592e3df83d0dcac1f74399e342ba99c718975d34b8a27e07eeca5_configure.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f5/98/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:f598fad5807fabde02cd16e5f7f26b486d0b5f87dec6ff3c3d5378ba68780ff1_install.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/e2/a4/sstate:quilt-native::0.67:r0::11:e2a41fd44aa7736bd318ec9c4a6b91e4b80563ba213035afe3df3cb96fdf7aaf_fetch.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/da/07/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:da07e6362c0b01fc2798456aa1319e2343a556e33d0857f90a129701eaa722b2_deploy_source_date_epoch.tar.zst.siginfo
> DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/72/12/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:7212eff5f191def566b82db976b37e1efcb0a8ccd964eff5b5682b651b7cb7fa_collect_spdx_deps.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/96/sstate:quilt-native::0.67:r0::11:7396e70fb1865087409e454eb8bacae3bcb4add6218b3cd707d2e96880b72936_populate_lic.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/9f/8e/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:9f8e9e5303322240b173483312d885144dfdf2228698868857a281b510ec3214_recipe_qa.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fd/c4/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:fdc40ea4bffb4485bdbb3c38a13b4e5d8d94d1c2cdb66c8659fe5f758a3530a3_populate_sysroot.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/ac/69/sstate:quilt-native::0.67:r0::11:ac69437e7d349349b5f1b2df79203d3ade24e59dc32f16cdf03e6dc92dd3fad6_unpack.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/94/9b/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:949b4472b14403843ae5219789e371979cbc9a53c7fa250033f55157186d902e_compile.tar.zst.siginfo
> DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/1b/da/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:1bdafcc9e6b4209bfbdf4b83f8b44e0cc3cc26cdeebe7d713a9f7fcba4b02d48_create_spdx.tar.zst.siginfo
> 
> and the sstate for quilt-native_0.67.bb:do_collect_spdx_deps is indeed missing, there is only x86:
> 
> $ ls /srv/autobuilder/autobuilder.yocto.io/pub/sstate/*/72/12/sstate\:quilt-native\:*collect_spdx_deps.tar.zst.siginfo 
> /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/72/12/sstate:quilt-native:x86_64-linux:0.67:r0:x86_64:11:7212eff5f191def566b82db976b37e1efcb0a8ccd964eff5b5682b651b7cb7fa_collect_spdx_deps.tar.zst.siginfo
> 
> The normal build works since it doesn't care about the siginfo files,
> just the actual sstate it needs and as a covered task, it just skips
> over this.
> 
> This highlights two issues:
> 
> a) Why is the single sstate siginfo file gone? The most likely
> explanation is that it wasn't accessed recently and was deleted by the
> cleanup/ageing code. This makes sense since the code in question never
> downloads it, only checks it exists and uses that as a marker. Our
> usage "accesses" are in a different codepath to the parsing check.
> 
> b) This means -S printdiff doesn't quite do what you'd expect since it
> doesn't skip "covered" tasks in the same way a normal build would.
> 
> To reproduce the test failure you should be able to just delete that
> siginfo file from your local sstate.

I had a look into b), where I needed to teach the runqueue differences
code about setscene covered tasks. A really hacky way to test that
would be:

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index bb999c9dc1..3592130046 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1697,7 +1697,7 @@ class RunQueue:
 
             tocheck.add(tid)
 
-        valid_new = self.validate_hashes(tocheck, self.cooker.data, 0, True, summary=False)
+        valid_new = self.validate_hashes(tocheck, self.cooker.data, 0, True, summary=True)
 
         # Tasks which are both setscene and noexec never care about dependencies
         # We therefore find tasks which are setscene and noexec and mark their
@@ -1720,8 +1722,21 @@ class RunQueue:
             if tid not in valid_new and tid not in noexec:
                 invalidtasks.add(tid)
 
+        self.sqdata = SQData()
+        self.sqdata.cantskip = set()
+        self.sqdata.tasks_scenequeue_done = set()
+        self.sqdata.sq_buildable = set()
+        build_scenequeue_data(self.sqdata, self.rqdata, None, self.cooker, None, self.sqdata)
+
+        for tid in self.rqdata.runq_setscene_tids:
+            if tid not in valid_new:
+                continue
+            for dep in self.sqdata.sq_covered_tasks[tid]:
+                if dep in invalidtasks:
+                    invalidtasks.remove(dep)
+
         found = set()
         for tid in invalidtasks:
             processed = set()
             toprocess = set([tid])
             while toprocess:
@@ -2961,6 +2978,8 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
                 sqrq.sq_deferred[tid] = sqdata.hashes[h]
                 bb.debug(1, "Deferring %s after %s" % (tid, sqdata.hashes[h]))
 
+    if not rq:
+        return
     update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True)
 
     # Compute a list of 'stale' sstate tasks where the current hash does not match the one


the code needing some proper refactoring to allow that for an
upstreamable patch. The trouble is that whilst it is improved, this
still showed glitches:

The differences between the current build and any cached tasks start at the following tasks:
/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/timezone/tzcode-native.bb:do_prepare_recipe_sysroot
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/makedevs/makedevs_1.0.1.bb:do_configure
/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/timezone/tzcode-native.bb:do_deploy_source_date_epoch
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/pigz/pigz_2.8.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/update-rc.d/update-rc.d_0.8.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_configure
/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_configure
/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/glibc/ldconfig-native_2.12.1.bb:do_configure
/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/gettext/gettext-minimal-native_0.22.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/zstd/zstd_1.5.5.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/unzip/unzip_6.0.bb:do_configure

e.g. why the tzcode-native.bb:do_deploy_source_date_epoch issue?

Further debugging with "bitbake tzcode-native -S printdiff -DDD":

DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/e2/a4/sstate:quilt-native::0.67:r0::11:e2a41fd44aa7736bd318ec9c4a6b91e4b80563ba213035afe3df3cb96fdf7aaf_fetch.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/b8/fb/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:b8fbbfbafeb7cdf333f1766d6b3c94822a21d42145e17377bede29e44a86a14d_prepare_recipe_sysroot.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/94/9b/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:949b4472b14403843ae5219789e371979cbc9a53c7fa250033f55157186d902e_compile.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/ee/ae/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:eeaecb5daeb66cd24558c8ad6f75a33cbc3a108d6c6e98ebcc2e28e73cc4978c_compile.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/83/6e/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:836eeaecc95081aa94d30ec4d61b30b325f92b9913c77190d0e22eeadb5b7d0b_create_runtime_spdx.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/9f/8e/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:9f8e9e5303322240b173483312d885144dfdf2228698868857a281b510ec3214_recipe_qa.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/59/02/sstate:tzcode-native::2023c:r0::11:5902de31682091449c8d40162f35f4ef0bee1a4879b3743b565504fd4fb03f27_populate_lic.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/03/07/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:0307b7910df3aae19e2714a290786fd46213b5664ed082eb38658ee4d0a7295b_recipe_qa.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/85/79/sstate:quilt-native::0.67:r0::11:85794fd6d83b61cb273e4d14f6e3581d7fa4ec743b455f1d21786cff032c262c_patch.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/a7/fa/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:a7fa933ded6a2c1c8a60de136a9a46cd1788fb47c53e5a00cb981349ffd260e5_configure.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/ac/69/sstate:quilt-native::0.67:r0::11:ac69437e7d349349b5f1b2df79203d3ade24e59dc32f16cdf03e6dc92dd3fad6_unpack.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fe/ae/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:feae04b2abf592e3df83d0dcac1f74399e342ba99c718975d34b8a27e07eeca5_configure.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/03/0d/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:030dba86bef371a20bb779cef7cdd93c0c84eaa0fadfe1d74d8784cfc9980b2c_deploy_source_date_epoch.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f5/98/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:f598fad5807fabde02cd16e5f7f26b486d0b5f87dec6ff3c3d5378ba68780ff1_install.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2b/cf/sstate:tzcode-native::2023c:r0::11:2bcf4112bf22bd356e5352e9f6dde17040e9f9ebc4a60b4321112f501a6596bd_fetch.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/9f/3b/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:9f3b9de096a3092769fc5c09a8f85de88aff486874b1c97150eb4cff252f5f03_install.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f6/97/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:f697fc1dffb4c3e7942e0a0e952978b50535cc61a8ed1bd1b0f5c9e1bc91d676_prepare_recipe_sysroot.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/66/4d/sstate:tzcode-native::2023c:r0::11:664de93a4c91a94e498ce6039aba6bd490699f18b4295163b78495524dda17b9_patch.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/91/da/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:91dadb158ecb0b8ac2178d8847bd8894bb8b95f84f53af47d617ef2c7eb2c3dd_populate_sysroot.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/b0/30/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:b030f98a1bbe000c1b9eda75252084fd2e43239d55d0b410306ca5566ecf422b_collect_spdx_deps.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/da/07/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:da07e6362c0b01fc2798456aa1319e2343a556e33d0857f90a129701eaa722b2_deploy_source_date_epoch.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/c2/92/sstate:tzcode-native::2023c:r0::11:c2925547a4521cb90a5e8b48c4e1397115800337248e1cd21ef25e5ab88e8834_unpack.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fd/c4/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:fdc40ea4bffb4485bdbb3c38a13b4e5d8d94d1c2cdb66c8659fe5f758a3530a3_populate_sysroot.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/37/9d/sstate:tzcode-native:aarch64-linux:2023c:r0:aarch64:11:379d43911b1203a6ff7232a38b8f1eb95bdc3df5924ee06ed4f3be84a691d864_create_spdx.tar.zst.siginfo

so it seems that missing siginfo of tasks *before* do_configure would
also cause the test to show issues.

I'm going to try a patch touching sstate files to try and improve
things. I do think the runqueue patches to improve printdiff may also
be worthwhile as the above was much more readable, which is the end
goal.

Cheers,

Richard