* [PATCH 1/2] sstate: Promote do_install to an sstate test
2020-11-01 12:53 [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work Richard Purdie
@ 2020-11-01 12:53 ` Richard Purdie
2020-11-01 12:53 ` [PATCH 2/2] selftest: Add binaryonly example and selftest Richard Purdie
2020-11-02 9:01 ` [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work Mikko Rapeli
2 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2020-11-01 12:53 UTC (permalink / raw)
To: openembedded-core; +Cc: Nicolas Dechesne, Viswanath Kraleti
Allow sstate to be generated for do_install tasks.
FIXME: Needs sstatesig rule mappings to be added for do_install as builds
from sstate currently install *everything* and shouldn't. do_install
rules are missing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
meta/classes/base.bbclass | 10 ++++++++++
meta/classes/staging.bbclass | 2 +-
meta/lib/oe/sstatesig.py | 2 +-
3 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 5a0b0c6b3e2..cb854ed6601 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -339,6 +339,14 @@ base_do_install() {
:
}
+SSTATETASKS += "do_install"
+do_install[sstate-plaindirs] = "${D}"
+
+python do_install_setscene () {
+ sstate_setscene(d)
+}
+addtask do_install_setscene
+
base_do_package() {
:
}
@@ -511,6 +519,8 @@ python () {
d.appendVarFlag('do_install', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
d.setVarFlag('do_install', 'fakeroot', '1')
d.setVarFlag('do_install', 'umask', '022')
+ d.setVarFlag('do_install_setscene', 'fakeroot', '1')
+ d.appendVarFlag('do_install_setscene', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
d.appendVarFlag('do_package', 'depends', ' virtual/fakeroot-native:do_populate_sysroot')
d.setVarFlag('do_package', 'fakeroot', '1')
d.setVarFlag('do_package', 'umask', '022')
diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index f0a619b35bd..11eadf18062 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -326,7 +326,7 @@ python extend_recipe_sysroot() {
# Create collapsed do_populate_sysroot -> do_populate_sysroot tree
for dep in taskdepdata:
data = setscenedeps[dep]
- if data[1] not in sstatetasks:
+ if data[1] not in sstatetasks or data[1] == "do_install":
for dep2 in setscenedeps:
data2 = setscenedeps[dep2]
if dep in data2[3]:
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 4ea29cbdfc3..15391255b83 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -480,7 +480,7 @@ def OEOuthashBasic(path, sigfile, task, d):
if "package_write_" in task or task == "package_qa":
include_owners = False
include_timestamps = False
- if task == "package":
+ if task == "package" or task == "install":
include_timestamps = d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1'
extra_content = d.getVar('HASHEQUIV_HASH_VERSION')
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH 2/2] selftest: Add binaryonly example and selftest
2020-11-01 12:53 [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work Richard Purdie
2020-11-01 12:53 ` [PATCH 1/2] sstate: Promote do_install to an sstate test Richard Purdie
@ 2020-11-01 12:53 ` Richard Purdie
2020-11-02 9:01 ` [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work Mikko Rapeli
2 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2020-11-01 12:53 UTC (permalink / raw)
To: openembedded-core; +Cc: Nicolas Dechesne, Viswanath Kraleti
This patch adds a selftest showing how locked sstate can be used to handle
a binary only delivery of some output.
xz is used as an example with a special "binary only" variant of the recipe
which is basically the standard one with some small tweaks so it has its own
recipe and package namespaces.
The test creates its own local sstate directory for the test so it doesn't write
to any main sstate store.
If the xz-binaryonly recipe is built with "binaryonly-secretsource" in BBPATH,
it functions as a normal recipe. The xz-binaryonly-src include file contains
the "secret" recipe build instructions such as where to get the source from
and how to configure it. There is a binaryonly-src bbclass which at the end
of do_install, writes out a locked sstate signatures file showing how to lock
the build to this specific binary sstate object.
If the xz-binaryonly recipe is built without any BBPATH additions but the sstate
signature lock include file is included, the build will look for and use the sstate
object specified and not run any of the configure/compile tasks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
.../xz-binaryonly-src.inc | 16 +++++++++
meta-selftest/classes/binaryonly-src.bbclass | 9 +++++
meta-selftest/classes/binaryonly.bbclass | 9 +++++
.../xz-binaryonly/xz-binaryonly_5.2.5.bb | 36 +++++++++++++++++++
meta/lib/oeqa/selftest/cases/binaryonly.py | 34 ++++++++++++++++++
5 files changed, 104 insertions(+)
create mode 100644 meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc
create mode 100644 meta-selftest/classes/binaryonly-src.bbclass
create mode 100644 meta-selftest/classes/binaryonly.bbclass
create mode 100644 meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb
create mode 100644 meta/lib/oeqa/selftest/cases/binaryonly.py
diff --git a/meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc b/meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc
new file mode 100644
index 00000000000..cd2294ba90e
--- /dev/null
+++ b/meta-selftest/binaryonly-secretsource/xz-binaryonly-src.inc
@@ -0,0 +1,16 @@
+LIC_FILES_CHKSUM = "file://COPYING;md5=97d554a32881fee0aa283d96e47cb24a \
+ file://COPYING.GPLv2;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
+ file://COPYING.GPLv3;md5=d32239bcb673463ab874e80d47fae504 \
+ file://COPYING.LGPLv2.1;md5=4fbd65380cdd255951079008b364516c \
+ file://lib/getopt.c;endline=23;md5=2069b0ee710572c03bb3114e4532cd84 \
+ "
+
+SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz"
+SRC_URI[md5sum] = "0d270c997aff29708c74d53f599ef717"
+SRC_URI[sha256sum] = "f6f4910fd033078738bd82bfba4f49219d03b17eb0794eb91efbae419f4aba10"
+UPSTREAM_CHECK_REGEX = "xz-(?P<pver>\d+(\.\d+)+)\.tar"
+
+CACHED_CONFIGUREVARS += "gl_cv_posix_shell=/bin/sh"
+
+inherit binaryonly-src
+
diff --git a/meta-selftest/classes/binaryonly-src.bbclass b/meta-selftest/classes/binaryonly-src.bbclass
new file mode 100644
index 00000000000..75bc5e2ca7b
--- /dev/null
+++ b/meta-selftest/classes/binaryonly-src.bbclass
@@ -0,0 +1,9 @@
+do_install[postfuncs] += "write_binaryonly_lockedsig"
+
+python write_binaryonly_lockedsig () {
+ outputfile = d.expand("${TOPDIR}/binaryonly-lockedsigs-demo.inc")
+
+ with open(outputfile, "a+") as f:
+ f.write('SIGGEN_LOCKEDSIGS_t-%s += "%s:do_install:%s"\n' % (d.getVar("PACKAGE_ARCH"), d.getVar("PN"), d.getVar("BB_TASKHASH")))
+ f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "t-%s"\n' % (d.getVar("MACHINE"), d.getVar("PACKAGE_ARCH")))
+}
diff --git a/meta-selftest/classes/binaryonly.bbclass b/meta-selftest/classes/binaryonly.bbclass
new file mode 100644
index 00000000000..4903c7cd934
--- /dev/null
+++ b/meta-selftest/classes/binaryonly.bbclass
@@ -0,0 +1,9 @@
+python () {
+ if not bb.data.inherits_class('binaryonly-src', d):
+ bb.build.deltask("do_fetch", d)
+ bb.build.deltask("do_unpack", d)
+ bb.build.deltask("do_patch", d)
+ #bb.build.deltask("do_prepare_recipe_sysroot", d)
+ #bb.build.deltask("do_configure", d)
+ #bb.build.deltask("do_compile", d)
+}
\ No newline at end of file
diff --git a/meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb b/meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb
new file mode 100644
index 00000000000..366c6e9751c
--- /dev/null
+++ b/meta-selftest/recipes-binaryonly/xz-binaryonly/xz-binaryonly_5.2.5.bb
@@ -0,0 +1,36 @@
+SUMMARY = "Utilities for managing LZMA compressed files"
+HOMEPAGE = "https://tukaani.org/xz/"
+SECTION = "base"
+
+BPN = "xz"
+
+# The source includes bits of PD, GPLv2, GPLv3, LGPLv2.1+, but the only file
+# which is GPLv3 is an m4 macro which isn't shipped in any of our packages,
+# and the LGPL bits are under lib/, which appears to be used for libgnu, which
+# appears to be used for DOS builds. So we're left with GPLv2+ and PD.
+LICENSE = "GPLv2+ & GPL-3.0-with-autoconf-exception & LGPLv2.1+ & PD"
+LICENSE_${PN} = "GPLv2+"
+LICENSE_${PN}-dev = "GPLv2+"
+LICENSE_${PN}-staticdev = "GPLv2+"
+LICENSE_${PN}-doc = "GPLv2+"
+LICENSE_${PN}-dbg = "GPLv2+"
+LICENSE_${PN}-locale = "GPLv2+"
+LICENSE_liblzma-bin = "PD"
+
+# We try to include the secret source but don't fail if we can't
+include xz-binaryonly-src.inc
+
+inherit binaryonly
+
+inherit autotools gettext
+
+PACKAGES =+ "liblzma-bin"
+DEBIAN_NOAUTONAME_liblzma-bin = "1"
+FILES_liblzma-bin = "${libdir}/liblzma*${SOLIBS}"
+
+inherit update-alternatives
+ALTERNATIVE_PRIORITY = "100"
+ALTERNATIVE_${PN} = "xz xzcat unxz \
+ lzma lzcat unlzma"
+
+BBCLASSEXTEND = "native nativesdk"
diff --git a/meta/lib/oeqa/selftest/cases/binaryonly.py b/meta/lib/oeqa/selftest/cases/binaryonly.py
new file mode 100644
index 00000000000..3901d41db0b
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/binaryonly.py
@@ -0,0 +1,34 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Author: Richard Purdie <richard.purdie@linuxfoundation.org>
+#
+
+import shutil
+import tempfile
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var, get_test_layer
+
+class BinaryOnly(OESelftestTestCase):
+ def test_binaryonly(self):
+ # We need to create our own local sstate directory but use the main sstate as a mirror for speed
+ tempdir = tempfile.mkdtemp(prefix='sstate_binaryonly')
+ testlayer = get_test_layer()
+ sstate_path = get_bb_var("SSTATE_DIR")
+ topdir = get_bb_var("TOPDIR")
+ self.append_config("SSTATE_DIR = \"%s\"" % tempdir)
+ self.append_config("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = 'warn'")
+ self.track_for_cleanup(tempdir)
+ self.append_config("SSTATE_MIRRORS += \"file://.* file:///%s/PATH\"" % sstate_path)
+
+ enable_source = "BBPATH .= ':%s/binaryonly-secretsource'" % testlayer
+ self.append_config(enable_source)
+ bitbake("xz-binaryonly -c clean")
+ bitbake("xz-binaryonly")
+ bitbake("xz-binaryonly -c clean")
+
+ enable_lockfile = "require ${TOPDIR}/binaryonly-lockedsigs-demo.inc"
+
+ self.remove_config(enable_source)
+ self.append_config(enable_lockfile)
+ bitbake("xz-binaryonly")
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work
2020-11-01 12:53 [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work Richard Purdie
2020-11-01 12:53 ` [PATCH 1/2] sstate: Promote do_install to an sstate test Richard Purdie
2020-11-01 12:53 ` [PATCH 2/2] selftest: Add binaryonly example and selftest Richard Purdie
@ 2020-11-02 9:01 ` Mikko Rapeli
2020-11-02 10:28 ` Richard Purdie
2 siblings, 1 reply; 6+ messages in thread
From: Mikko Rapeli @ 2020-11-02 9:01 UTC (permalink / raw)
To: richard.purdie; +Cc: openembedded-core, nicolas.dechesne, vkraleti
Hi,
On Sun, Nov 01, 2020 at 12:53:00PM +0000, Richard Purdie wrote:
> At the Yocto Project Summit, there was some talk about how prebuilt binaries could
> be better supported. There was some discussion about how "locked sstate" could
> actually handle this quite neatly. I thought I'd have a look at that and came up
> with a few patches to illustrate what I was talking about.
Interesting, thanks for bringing this up also on mailing list!
How could one implement a variant where source recipe vs. pre-compiled
binary build would be chosen based on access rights to a git tree in
SRC_URI?
Cheers,
-Mikko
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work
2020-11-02 9:01 ` [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work Mikko Rapeli
@ 2020-11-02 10:28 ` Richard Purdie
2020-11-02 11:48 ` Mikko Rapeli
0 siblings, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2020-11-02 10:28 UTC (permalink / raw)
To: Mikko.Rapeli; +Cc: openembedded-core, nicolas.dechesne, vkraleti
On Mon, 2020-11-02 at 09:01 +0000, Mikko.Rapeli@bmw.de wrote:
> Hi,
>
> On Sun, Nov 01, 2020 at 12:53:00PM +0000, Richard Purdie wrote:
> > At the Yocto Project Summit, there was some talk about how prebuilt
> > binaries could
> > be better supported. There was some discussion about how "locked
> > sstate" could
> > actually handle this quite neatly. I thought I'd have a look at
> > that and came up
> > with a few patches to illustrate what I was talking about.
>
> Interesting, thanks for bringing this up also on mailing list!
>
> How could one implement a variant where source recipe vs. pre-
> compiled
> binary build would be chosen based on access rights to a git tree in
> SRC_URI?
That gets hard since it would have to determine that at parse time. If
you have to check that for each repository, its going to slow things
down quite a bit. It would end up needing to be written into
immediately expanded python which set a variable to switch between the
include files.
Definitely possible, particularly if you write out some kind of cache
but its not going to be particularly neat.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core] [PATCH 0/2] Example of how prebuilt binaries and locked sstate could work
2020-11-02 10:28 ` Richard Purdie
@ 2020-11-02 11:48 ` Mikko Rapeli
0 siblings, 0 replies; 6+ messages in thread
From: Mikko Rapeli @ 2020-11-02 11:48 UTC (permalink / raw)
To: richard.purdie; +Cc: openembedded-core, nicolas.dechesne, vkraleti
On Mon, Nov 02, 2020 at 10:28:53AM +0000, Richard Purdie wrote:
> On Mon, 2020-11-02 at 09:01 +0000, Mikko.Rapeli@bmw.de wrote:
> > Hi,
> >
> > On Sun, Nov 01, 2020 at 12:53:00PM +0000, Richard Purdie wrote:
> > > At the Yocto Project Summit, there was some talk about how prebuilt
> > > binaries could
> > > be better supported. There was some discussion about how "locked
> > > sstate" could
> > > actually handle this quite neatly. I thought I'd have a look at
> > > that and came up
> > > with a few patches to illustrate what I was talking about.
> >
> > Interesting, thanks for bringing this up also on mailing list!
> >
> > How could one implement a variant where source recipe vs. pre-
> > compiled
> > binary build would be chosen based on access rights to a git tree in
> > SRC_URI?
>
> That gets hard since it would have to determine that at parse time. If
> you have to check that for each repository, its going to slow things
> down quite a bit. It would end up needing to be written into
> immediately expanded python which set a variable to switch between the
> include files.
>
> Definitely possible, particularly if you write out some kind of cache
> but its not going to be particularly neat.
Yep, so doing a check before bitbake build and then setting a local.conf
variable based on result is what would work then, and is what we do.
Another detail worth mentioning is the use binary packages generated from source
build which only get fetched and extracted in binary build.
Cheers,
-Mikko
^ permalink raw reply [flat|nested] 6+ messages in thread