[PATCH 0/2] Example of how prebuilt binaries and locked sstate could work

[PATCH 0/2] Example of how prebuilt binaries and locked sstate could work

[Richard Purdie]

At the Yocto Project Summit, there was some talk about how prebuilt binaries could
be better supported. There was some discussion about how "locked sstate" could 
actually handle this quite neatly. I thought I'd have a look at that and came up 
with a few patches to illustrate what I was talking about.

Rather than talk about this in abstract terms, I've written this up as an oeqa
selftest, you can run it as "oe-selftest -r binaryonly". The selftest patch commit
message contains information about the example "binaryonly" implementation.

This series is a proof of concept and there are a list of known issues:

This patch series depends upon:
 * the libxcb owner/group fix I sent out
 * a bitbake patch to siggen in bitbake master-next to remove what I think is a 
   misfunctioning optimisation to get_unihash
 * a fix to sstatesig in OE-Core master-next to adapt to the above bitbake fix

This series contains a patch to enable sstate for do_install globally. As noted
in the patch, whilst it builds, there are some rules missing to make it work 
optimially.

There is a bug in the way locked hashes were being handled by dependent tasks. The
two patches in master-next mentioned above are attempts to try and fix that, they 
certainly make this selftest work but I have concerns they may break something else.
I'm therefore performing wider autobuilder testing to see if that shows any issues.

The proof of concept here isn't 100% complete. In particular:

* We need a more generic/usable way to lock/unlock specific tasks in recipes,
  the test just generates a file of the right format for now to show how it works.
* A warning is shown for the locked recipe. We need a way to specify not to warn
  for a specific list of recipes.
* If the sstate object the recipe is locked to is unavailable it will currently
  fail the build obtusely. We need to write some kind of user readable error.
* There is no mechanism to ship specific sstate within a layer, we'd probably
  need one for this. The existing mirror syntax will probably help/work.
* The source date epoch and populate_lic tasks need to be properly handled, I've
  just ignored them in this demo.

I'd note that the BBPATH trick to enable/disable the source build could be done
as separate layers, I just didn't want to add multiple layers to OE-Core just for
this. The reason for this trick is that in many cases, the URLs and build process
for the source needs to be kept separate and not disclosed. That source inc is 
therefore designed to be seperable.

Richard Purdie (2):
  sstate: Promote do_install to an sstate test
  selftest: Add binaryonly example and selftest

 .../xz-binaryonly-src.inc                     | 16 +++++++++
 meta-selftest/classes/binaryonly-src.bbclass  |  9 +++++
 meta-selftest/classes/binaryonly.bbclass      |  9 +++++
 .../xz-binaryonly/xz-binaryonly_5.2.5.bb      | 36 +++++++++++++++++++
 meta/classes/base.bbclass                     | 10 ++++++
 meta/classes/staging.bbclass                  |  2 +-
 meta/lib/oe/sstatesig.py                      |  2 +-
 meta/lib/oeqa/selftest/cases/binaryonly.py    | 34 ++++++++++++++++++
 8 files changed, 116 insertions(+), 2 deletions(-)
 create mode 100644 meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc
 create mode 100644 meta-selftest/classes/binaryonly-src.bbclass
 create mode 100644 meta-selftest/classes/binaryonly.bbclass
 create mode 100644 meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/binaryonly.py

-- 
2.25.1

[PATCH 1/2] sstate: Promote do_install to an sstate test

[Richard Purdie]

Allow sstate to be generated for do_install tasks.

FIXME: Needs sstatesig rule mappings to be added for do_install as builds
from sstate currently install *everything* and shouldn't. do_install
rules are missing.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/base.bbclass    | 10 ++++++++++
 meta/classes/staging.bbclass |  2 +-
 meta/lib/oe/sstatesig.py     |  2 +-
 3 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 5a0b0c6b3e2..cb854ed6601 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -339,6 +339,14 @@ base_do_install() {
 	:
 }
 
+SSTATETASKS += "do_install"
+do_install[sstate-plaindirs] = "${D}"
+
+python do_install_setscene () {
+    sstate_setscene(d)
+}
+addtask do_install_setscene
+
 base_do_package() {
 	:
 }
@@ -511,6 +519,8 @@ python () {
         d.appendVarFlag('do_install', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
         d.setVarFlag('do_install', 'fakeroot', '1')
         d.setVarFlag('do_install', 'umask', '022')
+        d.setVarFlag('do_install_setscene', 'fakeroot', '1')
+        d.appendVarFlag('do_install_setscene', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
         d.appendVarFlag('do_package', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
         d.setVarFlag('do_package', 'fakeroot', '1')
         d.setVarFlag('do_package', 'umask', '022')
diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index f0a619b35bd..11eadf18062 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -326,7 +326,7 @@ python extend_recipe_sysroot() {
     # Create collapsed do_populate_sysroot -> do_populate_sysroot tree
     for dep in taskdepdata:
         data = setscenedeps[dep]
-        if data[1] not in sstatetasks:
+        if data[1] not in sstatetasks or data[1] == "do_install":
             for dep2 in setscenedeps:
                 data2 = setscenedeps[dep2]
                 if dep in data2[3]:
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 4ea29cbdfc3..15391255b83 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -480,7 +480,7 @@ def OEOuthashBasic(path, sigfile, task, d):
     if "package_write_" in task or task == "package_qa":
         include_owners = False
     include_timestamps = False
-    if task == "package":
+    if task == "package" or task == "install":
         include_timestamps = d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1'
     extra_content = d.getVar('HASHEQUIV_HASH_VERSION')
 
-- 
2.25.1

[PATCH 2/2] selftest: Add binaryonly example and selftest

[Richard Purdie]

This patch adds a selftest showing how locked sstate can be used to handle
a binary only delivery of some output.

xz is used as an example with a special "binary only" variant of the recipe
which is basically the standard one with some small tweaks so it has its own
recipe and package namespaces.

The test creates its own local sstate directory for the test so it doesn't write
to any main sstate store.

If the xz-binaryonly recipe is built with "binaryonly-secretsource" in BBPATH,
it functions as a normal recipe. The xz-binaryonly-src include file contains
the "secret" recipe build instructions such as where to get the source from
and how to configure it. There is a binaryonly-src bbclass which at the end
of do_install, writes out a locked sstate signatures file showing how to lock
the build to this specific binary sstate object.

If the xz-binaryonly recipe is built without any BBPATH additions but the sstate
signature lock include file is included, the build will look for and use the sstate
object specified and not run any of the configure/compile tasks.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../xz-binaryonly-src.inc                     | 16 +++++++++
 meta-selftest/classes/binaryonly-src.bbclass  |  9 +++++
 meta-selftest/classes/binaryonly.bbclass      |  9 +++++
 .../xz-binaryonly/xz-binaryonly_5.2.5.bb      | 36 +++++++++++++++++++
 meta/lib/oeqa/selftest/cases/binaryonly.py    | 34 ++++++++++++++++++
 5 files changed, 104 insertions(+)
 create mode 100644 meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc
 create mode 100644 meta-selftest/classes/binaryonly-src.bbclass
 create mode 100644 meta-selftest/classes/binaryonly.bbclass
 create mode 100644 meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/binaryonly.py

diff --git a/meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc b/meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc
new file mode 100644
index 00000000000..cd2294ba90e
--- /dev/null
+++ b/meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc
@@ -0,0 +1,16 @@
+LIC_FILES_CHKSUM = "file://COPYING;md5=97d554a32881fee0aa283d96e47cb24a \
+                    file://COPYING.GPLv2;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
+                    file://COPYING.GPLv3;md5=d32239bcb673463ab874e80d47fae504 \
+                    file://COPYING.LGPLv2.1;md5=4fbd65380cdd255951079008b364516c \
+                    file://lib/getopt.c;endline=23;md5=2069b0ee710572c03bb3114e4532cd84 \
+                    "
+
+SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz"
+SRC_URI[md5sum] = "0d270c997aff29708c74d53f599ef717"
+SRC_URI[sha256sum] = "f6f4910fd033078738bd82bfba4f49219d03b17eb0794eb91efbae419f4aba10"
+UPSTREAM_CHECK_REGEX = "xz-(?P<pver>\d+(\.\d+)+)\.tar"
+
+CACHED_CONFIGUREVARS += "gl_cv_posix_shell=/bin/sh"
+
+inherit binaryonly-src
+
diff --git a/meta-selftest/classes/binaryonly-src.bbclass b/meta-selftest/classes/binaryonly-src.bbclass
new file mode 100644
index 00000000000..75bc5e2ca7b
--- /dev/null
+++ b/meta-selftest/classes/binaryonly-src.bbclass
@@ -0,0 +1,9 @@
+do_install[postfuncs] += "write_binaryonly_lockedsig"
+
+python write_binaryonly_lockedsig () {
+    outputfile = d.expand("${TOPDIR}/binaryonly-lockedsigs-demo.inc")
+
+    with open(outputfile, "a+") as f:
+        f.write('SIGGEN_LOCKEDSIGS_t-%s += "%s:do_install:%s"\n' % (d.getVar("PACKAGE_ARCH"), d.getVar("PN"), d.getVar("BB_TASKHASH")))
+        f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "t-%s"\n' % (d.getVar("MACHINE"), d.getVar("PACKAGE_ARCH")))
+}
diff --git a/meta-selftest/classes/binaryonly.bbclass b/meta-selftest/classes/binaryonly.bbclass
new file mode 100644
index 00000000000..4903c7cd934
--- /dev/null
+++ b/meta-selftest/classes/binaryonly.bbclass
@@ -0,0 +1,9 @@
+python () {
+    if not bb.data.inherits_class('binaryonly-src', d):
+        bb.build.deltask("do_fetch", d)
+        bb.build.deltask("do_unpack", d)
+        bb.build.deltask("do_patch", d)
+        #bb.build.deltask("do_prepare_recipe_sysroot", d)
+        #bb.build.deltask("do_configure", d)
+        #bb.build.deltask("do_compile", d)
+}
\ No newline at end of file
diff --git a/meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb b/meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb
new file mode 100644
index 00000000000..366c6e9751c
--- /dev/null
+++ b/meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb
@@ -0,0 +1,36 @@
+SUMMARY = "Utilities for managing LZMA compressed files"
+HOMEPAGE = "https://tukaani.org/xz/"
+SECTION = "base"
+
+BPN = "xz"
+
+# The source includes bits of PD, GPLv2, GPLv3, LGPLv2.1+, but the only file
+# which is GPLv3 is an m4 macro which isn't shipped in any of our packages,
+# and the LGPL bits are under lib/, which appears to be used for libgnu, which
+# appears to be used for DOS builds. So we're left with GPLv2+ and PD.
+LICENSE = "GPLv2+ & GPL-3.0-with-autoconf-exception & LGPLv2.1+ & PD"
+LICENSE_${PN} = "GPLv2+"
+LICENSE_${PN}-dev = "GPLv2+"
+LICENSE_${PN}-staticdev = "GPLv2+"
+LICENSE_${PN}-doc = "GPLv2+"
+LICENSE_${PN}-dbg = "GPLv2+"
+LICENSE_${PN}-locale = "GPLv2+"
+LICENSE_liblzma-bin = "PD"
+
+# We try to include the secret source but don't fail if we can't
+include xz-binaryonly-src.inc
+
+inherit binaryonly
+
+inherit autotools gettext
+
+PACKAGES =+ "liblzma-bin"
+DEBIAN_NOAUTONAME_liblzma-bin = "1"
+FILES_liblzma-bin = "${libdir}/liblzma*${SOLIBS}"
+
+inherit update-alternatives
+ALTERNATIVE_PRIORITY = "100"
+ALTERNATIVE_${PN} = "xz xzcat unxz \
+                     lzma lzcat unlzma"
+
+BBCLASSEXTEND = "native nativesdk"
diff --git a/meta/lib/oeqa/selftest/cases/binaryonly.py b/meta/lib/oeqa/selftest/cases/binaryonly.py
new file mode 100644
index 00000000000..3901d41db0b
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/binaryonly.py
@@ -0,0 +1,34 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Author: Richard Purdie <richard.purdie@linuxfoundation.org>
+#
+
+import shutil
+import tempfile
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var, get_test_layer
+
+class BinaryOnly(OESelftestTestCase):
+    def test_binaryonly(self):
+        # We need to create our own local sstate directory but use the main sstate as a mirror for speed
+        tempdir = tempfile.mkdtemp(prefix='sstate_binaryonly')
+        testlayer = get_test_layer()
+        sstate_path = get_bb_var("SSTATE_DIR")
+        topdir = get_bb_var("TOPDIR")
+        self.append_config("SSTATE_DIR = \"%s\"" % tempdir)
+        self.append_config("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = 'warn'")
+        self.track_for_cleanup(tempdir)
+        self.append_config("SSTATE_MIRRORS += \"file://.* file:///%s/PATH\"" % sstate_path)
+
+        enable_source = "BBPATH .= ':%s/binaryonly-secretsource'" % testlayer
+        self.append_config(enable_source)
+        bitbake("xz-binaryonly -c clean")
+        bitbake("xz-binaryonly")
+        bitbake("xz-binaryonly -c clean")
+
+        enable_lockfile = "require ${TOPDIR}/binaryonly-lockedsigs-demo.inc"
+
+        self.remove_config(enable_source)
+        self.append_config(enable_lockfile)
+        bitbake("xz-binaryonly")
-- 
2.25.1

Re: [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work

[Mikko Rapeli]

Hi,

On Sun, Nov 01, 2020 at 12:53:00PM +0000, Richard Purdie wrote:
> At the Yocto Project Summit, there was some talk about how prebuilt binaries could
> be better supported. There was some discussion about how "locked sstate" could 
> actually handle this quite neatly. I thought I'd have a look at that and came up 
> with a few patches to illustrate what I was talking about.

Interesting, thanks for bringing this up also on mailing list!

How could one implement a variant where source recipe vs. pre-compiled
binary build would be chosen based on access rights to a git tree in
SRC_URI?

Cheers,

-Mikko

Re: [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work

[Richard Purdie]

On Mon, 2020-11-02 at 09:01 +0000, Mikko.Rapeli@bmw.de wrote:
> Hi,
> 
> On Sun, Nov 01, 2020 at 12:53:00PM +0000, Richard Purdie wrote:
> > At the Yocto Project Summit, there was some talk about how prebuilt
> > binaries could
> > be better supported. There was some discussion about how "locked
> > sstate" could 
> > actually handle this quite neatly. I thought I'd have a look at
> > that and came up 
> > with a few patches to illustrate what I was talking about.
> 
> Interesting, thanks for bringing this up also on mailing list!
> 
> How could one implement a variant where source recipe vs. pre-
> compiled
> binary build would be chosen based on access rights to a git tree in
> SRC_URI?

That gets hard since it would have to determine that at parse time. If
you have to check that for each repository, its going to slow things
down quite a bit. It would end up needing to be written into
immediately expanded python which set a variable to switch between the
include files.

Definitely possible, particularly if you write out some kind of cache
but its not going to be particularly neat.

Cheers,

Richard

Re: [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work

[Mikko Rapeli]

On Mon, Nov 02, 2020 at 10:28:53AM +0000, Richard Purdie wrote:
> On Mon, 2020-11-02 at 09:01 +0000, Mikko.Rapeli@bmw.de wrote:
> > Hi,
> > 
> > On Sun, Nov 01, 2020 at 12:53:00PM +0000, Richard Purdie wrote:
> > > At the Yocto Project Summit, there was some talk about how prebuilt
> > > binaries could
> > > be better supported. There was some discussion about how "locked
> > > sstate" could 
> > > actually handle this quite neatly. I thought I'd have a look at
> > > that and came up 
> > > with a few patches to illustrate what I was talking about.
> > 
> > Interesting, thanks for bringing this up also on mailing list!
> > 
> > How could one implement a variant where source recipe vs. pre-
> > compiled
> > binary build would be chosen based on access rights to a git tree in
> > SRC_URI?
> 
> That gets hard since it would have to determine that at parse time. If
> you have to check that for each repository, its going to slow things
> down quite a bit. It would end up needing to be written into
> immediately expanded python which set a variable to switch between the
> include files.
> 
> Definitely possible, particularly if you write out some kind of cache
> but its not going to be particularly neat.

Yep, so doing a check before bitbake build and then setting a local.conf
variable based on result is what would work then, and is what we do.

Another detail worth mentioning is the use binary packages generated from source
build which only get fetched and extracted in binary build.

Cheers,

-Mikko