* [PATCH 0/3] Add initial container IMAGE_FSTYPES support
@ 2017-01-18 18:31 Randy Witt
2017-01-18 18:31 ` [PATCH 1/3] image-container.bbclass: Add the "container" IMAGE_FSTYPES Randy Witt
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Randy Witt @ 2017-01-18 18:31 UTC (permalink / raw)
To: openembedded-core
The following changes since commit 63f899a950daf1018999455bafa7a2be8b22f164:
bitbake: toaster: bin/toaster whitelist TOASTER_DIR (2017-01-17 13:18:47 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib rewitt/contwip
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/contwip
Randy Witt (3):
image-container.bbclass: Add the "container" IMAGE_FSTYPES
image-container.bbclass: Error if not using linux-dummy
selftest/containerimage.py: Add container IMAGE_FSTYPES test
.../container-image/container-image-testpkg.bb | 8 +++
.../container-image/container-test-image.bb | 8 +++
meta/classes/image-container.bbclass | 21 ++++++++
meta/classes/image.bbclass | 3 ++
meta/classes/image_types.bbclass | 1 +
meta/lib/oeqa/selftest/containerimage.py | 57 ++++++++++++++++++++++
6 files changed, 98 insertions(+)
create mode 100644 meta-selftest/recipes-test/container-image/container-image-testpkg.bb
create mode 100644 meta-selftest/recipes-test/container-image/container-test-image.bb
create mode 100644 meta/classes/image-container.bbclass
create mode 100644 meta/lib/oeqa/selftest/containerimage.py
--
2.9.3
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH 1/3] image-container.bbclass: Add the "container" IMAGE_FSTYPES 2017-01-18 18:31 [PATCH 0/3] Add initial container IMAGE_FSTYPES support Randy Witt @ 2017-01-18 18:31 ` Randy Witt 2017-01-18 18:31 ` [PATCH 2/3] image-container.bbclass: Error if not using linux-dummy Randy Witt 2017-01-18 18:31 ` [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test Randy Witt 2 siblings, 0 replies; 7+ messages in thread From: Randy Witt @ 2017-01-18 18:31 UTC (permalink / raw) To: openembedded-core The "container" fstype does very little other than pick tar.bz2 as the actual image type and disable installation of ROOTFS_BOOTSTRAP_INSTALL. [YOCTO #9502] Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> --- meta/classes/image-container.bbclass | 3 +++ meta/classes/image.bbclass | 3 +++ meta/classes/image_types.bbclass | 1 + 3 files changed, 7 insertions(+) create mode 100644 meta/classes/image-container.bbclass diff --git a/meta/classes/image-container.bbclass b/meta/classes/image-container.bbclass new file mode 100644 index 0000000..a10ee32 --- /dev/null +++ b/meta/classes/image-container.bbclass @@ -0,0 +1,3 @@ +ROOTFS_BOOTSTRAP_INSTALL = "" +IMAGE_TYPES_MASKED += "container" +IMAGE_TYPEDEP_container = "tar.bz2" diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 04fd5f9..b2c2a28 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -144,6 +144,9 @@ inherit ${IMAGE_TYPE_live} IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", "qcow2", "hdddirect"], "image-vm", "", d)}' inherit ${IMAGE_TYPE_vm} +IMAGE_TYPE_container = '${@bb.utils.contains("IMAGE_FSTYPES", "container", "image-container", "", d)}' +inherit ${IMAGE_TYPE_container} + def build_uboot(d): if 'u-boot' in (d.getVar('IMAGE_FSTYPES') or ''): return "image_types_uboot" diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass index 7748dee..727a87e 100644 --- a/meta/classes/image_types.bbclass +++ b/meta/classes/image_types.bbclass @@ -328,6 +328,7 @@ IMAGE_TYPES = " \ hdddirect \ elf \ wic wic.gz wic.bz2 wic.lzma \ + container \ " # Compression is a special case of conversion. The old variable -- 2.9.3 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/3] image-container.bbclass: Error if not using linux-dummy 2017-01-18 18:31 [PATCH 0/3] Add initial container IMAGE_FSTYPES support Randy Witt 2017-01-18 18:31 ` [PATCH 1/3] image-container.bbclass: Add the "container" IMAGE_FSTYPES Randy Witt @ 2017-01-18 18:31 ` Randy Witt 2017-01-18 18:31 ` [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test Randy Witt 2 siblings, 0 replies; 7+ messages in thread From: Randy Witt @ 2017-01-18 18:31 UTC (permalink / raw) To: openembedded-core In the typical case of a container, the user doesn't particularly care about the kernel version used. Since using linux-dummy prevents building a kernel in the most common case, alert the user if they choose a container image, but don't also specify linux-dummy. In the cases the user actually does want to build against particular kernel headers, etc, they can override by setting IMAGE_CONTAINER_NO_DUMMY = "1". [YOCTO #9502] Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> --- meta/classes/image-container.bbclass | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/meta/classes/image-container.bbclass b/meta/classes/image-container.bbclass index a10ee32..f002858 100644 --- a/meta/classes/image-container.bbclass +++ b/meta/classes/image-container.bbclass @@ -1,3 +1,21 @@ ROOTFS_BOOTSTRAP_INSTALL = "" IMAGE_TYPES_MASKED += "container" IMAGE_TYPEDEP_container = "tar.bz2" + +python __anonymous() { + if "container" in d.getVar("IMAGE_FSTYPES") and \ + d.getVar("IMAGE_CONTAINER_NO_DUMMY") != "1" and \ + "linux-dummy" not in d.getVar("PREFERRED_PROVIDER_virtual/kernel"): + msg = '"container" is in IMAGE_FSTYPES, but ' \ + 'PREFERRED_PROVIDER_virtual/kernel is not "linux-dummy". ' \ + 'Unless a particular kernel is needed, using linux-dummy will ' \ + 'prevent a kernel from being built, which can reduce ' \ + 'build times. If you don\'t want to use "linux-dummy", set ' \ + '"IMAGE_CONTAINER_NO_DUMMY" to "1".' + + # Raising skip recipe was Paul's clever idea. It causes the error to + # only be shown for the recipes actually requested to build, rather + # than bb.fatal which would appear for all recipes inheriting the + # class. + raise bb.parse.SkipRecipe(msg) +} -- 2.9.3 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test 2017-01-18 18:31 [PATCH 0/3] Add initial container IMAGE_FSTYPES support Randy Witt 2017-01-18 18:31 ` [PATCH 1/3] image-container.bbclass: Add the "container" IMAGE_FSTYPES Randy Witt 2017-01-18 18:31 ` [PATCH 2/3] image-container.bbclass: Error if not using linux-dummy Randy Witt @ 2017-01-18 18:31 ` Randy Witt 2017-01-19 12:01 ` Burton, Ross 2017-01-19 21:52 ` [PATCH] " Randy Witt 2 siblings, 2 replies; 7+ messages in thread From: Randy Witt @ 2017-01-18 18:31 UTC (permalink / raw) To: openembedded-core This test checks to make sure only the files expected exist in a container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to all images without the user specifying it. But this test should help if a developer in the future ever silently adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can make sure it also gets removed from a container image. [YOCTO #9502] Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> --- .../container-image/container-image-testpkg.bb | 8 +++ .../container-image/container-test-image.bb | 8 +++ meta/lib/oeqa/selftest/containerimage.py | 57 ++++++++++++++++++++++ 3 files changed, 73 insertions(+) create mode 100644 meta-selftest/recipes-test/container-image/container-image-testpkg.bb create mode 100644 meta-selftest/recipes-test/container-image/container-test-image.bb create mode 100644 meta/lib/oeqa/selftest/containerimage.py diff --git a/meta-selftest/recipes-test/container-image/container-image-testpkg.bb b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb new file mode 100644 index 0000000..42014dd --- /dev/null +++ b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb @@ -0,0 +1,8 @@ +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +do_install_append() { + install -d ${D}${base_bindir} + touch ${D}${base_bindir}/theapp +} diff --git a/meta-selftest/recipes-test/container-image/container-test-image.bb b/meta-selftest/recipes-test/container-image/container-test-image.bb new file mode 100644 index 0000000..d5f939c --- /dev/null +++ b/meta-selftest/recipes-test/container-image/container-test-image.bb @@ -0,0 +1,8 @@ +IMAGE_INSTALL += "container-image-testpkg" + +LICENSE = "MIT" + +IMAGE_FSTYPES = "container" +IMAGE_LINGUAS = "" + +inherit core-image diff --git a/meta/lib/oeqa/selftest/containerimage.py b/meta/lib/oeqa/selftest/containerimage.py new file mode 100644 index 0000000..54e792a --- /dev/null +++ b/meta/lib/oeqa/selftest/containerimage.py @@ -0,0 +1,57 @@ +import os + +from oeqa.selftest.base import oeSelfTest +from oeqa.utils.commands import bitbake, runCmd + +# This test builds an image with using the "container" IMAGE_FSTYPE, and +# ensures that then files in the image are only the ones expected. +# +# The only package added to the image is container_image_testpkg, which +# contains one file. However, due to some other things not cleaning up during +# rootfs creation, there is some kruft. Ideally bugs will be filed and the +# kruft removed, but for now we whitelist some known set. +# +# Also for performance reason we're only checking the kruft when using ipk. +# When using deb, and rpm it is a bit different and we could test all +# of them, but this test is more to catch if other packages get added by +# default other than what is in ROOTFS_BOOTSTRAP_INSTALL. +# +class ContainerImageTests(oeSelfTest): + + # Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar" that + # the conversion type bar gets added as a dep as well + def test_expected_files(self): + + expected_files = [ + './', + './bin/', + './bin/theapp', + './etc/', + './etc/default/', + './etc/default/postinst', + './etc/ld.so.cache', + './etc/timestamp', + './etc/version', + './run/', + './var/', + './var/cache/', + './var/cache/ldconfig/', + './var/cache/ldconfig/aux-cache', + './var/cache/opkg/', + './var/lib/', + './var/lib/opkg/' + ] + + self.write_config('PREFERRED_PROVIDER_virtual/kernel = "linux-dummy"') + self.append_config('MACHINE = "qemux86"') + self.append_config('IMAGE_FSTYPES = "container"') + self.append_config('PACKAGE_CLASSES = "package_ipk"') + + # Build the image of course + bitbake('container-test-image') + + image = os.path.join(self.builddir, "tmp/deploy/images/qemux86/container-test-image-qemux86.tar.bz2") + + # Ensure the files in the image are what we expect + result = runCmd("tar tf {} | sort".format(image), shell=True) + self.assertEqual(result.output.split('\n'), expected_files) -- 2.9.3 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test 2017-01-18 18:31 ` [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test Randy Witt @ 2017-01-19 12:01 ` Burton, Ross 2017-01-19 22:50 ` Randy Witt 2017-01-19 21:52 ` [PATCH] " Randy Witt 1 sibling, 1 reply; 7+ messages in thread From: Burton, Ross @ 2017-01-19 12:01 UTC (permalink / raw) To: Randy Witt; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 1344 bytes --] On 18 January 2017 at 18:31, Randy Witt <randy.e.witt@linux.intel.com> wrote: > + expected_files = [ > + './', > + './bin/', > + './bin/theapp', > + './etc/', > + './etc/default/', > + './etc/default/postinst', > + './etc/ld.so.cache', > + './etc/timestamp', > + './etc/version', > + './run/', > + './var/', > + './var/cache/', > + './var/cache/ldconfig/', > + './var/cache/ldconfig/aux-cache', > + './var/cache/opkg/', > + './var/lib/', > + './var/lib/opkg/' > + ] > > These should use ${bindir} etc so they're resilient to distros changing the paths (please use get_bb_vars() to only invoke bitbake once). + image = os.path.join(self.builddir, "tmp/deploy/images/qemux86/ > container-test-image-qemux86.tar.bz2") > Don't hardcode this path, use ${DEPLOY_DIR_IMAGE} (fetch it with the paths above for speed) which will get you to tmp/deploy/images/[machine]. Again, these paths can change (poky sets TMPDIR to tmp, nodistro uses tmp-${TCLIBC}). Ross [-- Attachment #2: Type: text/html, Size: 2337 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test 2017-01-19 12:01 ` Burton, Ross @ 2017-01-19 22:50 ` Randy Witt 0 siblings, 0 replies; 7+ messages in thread From: Randy Witt @ 2017-01-19 22:50 UTC (permalink / raw) To: Burton, Ross; +Cc: OE-core On 01/19/2017 04:01 AM, Burton, Ross wrote: > On 18 January 2017 at 18:31, Randy Witt <randy.e.witt@linux.intel.com> > wrote: > >> + expected_files = [ >> + './', >> + './bin/', >> + './bin/theapp', >> + './etc/', >> + './etc/default/', >> + './etc/default/postinst', >> + './etc/ld.so.cache', >> + './etc/timestamp', >> + './etc/version', >> + './run/', >> + './var/', >> + './var/cache/', >> + './var/cache/ldconfig/', >> + './var/cache/ldconfig/aux-cache', >> + './var/cache/opkg/', >> + './var/lib/', >> + './var/lib/opkg/' >> + ] >> >> These should use ${bindir} etc so they're resilient to distros changing > the paths (please use get_bb_vars() to only invoke bitbake once). > > + image = os.path.join(self.builddir, "tmp/deploy/images/qemux86/ >> container-test-image-qemux86.tar.bz2") >> > > Don't hardcode this path, use ${DEPLOY_DIR_IMAGE} (fetch it with the paths > above for speed) which will get you to tmp/deploy/images/[machine]. Again, > these paths can change (poky sets TMPDIR to tmp, nodistro uses > tmp-${TCLIBC}). > > Ross > I just sent a v2, but screwed up the subject. I'll resend the whole series again even though only 3/3 changed, to hopefully make it easier on you. ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH] selftest/containerimage.py: Add container IMAGE_FSTYPES test 2017-01-18 18:31 ` [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test Randy Witt 2017-01-19 12:01 ` Burton, Ross @ 2017-01-19 21:52 ` Randy Witt 1 sibling, 0 replies; 7+ messages in thread From: Randy Witt @ 2017-01-19 21:52 UTC (permalink / raw) To: openembedded-core This test checks to make sure only the files expected exist in a container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to all images without the user specifying it. But this test should help if a developer in the future ever silently adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can make sure it also gets removed from a container image. [YOCTO #9502] Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> --- .../container-image/container-image-testpkg.bb | 8 +++ .../container-image/container-test-image.bb | 8 +++ meta/lib/oeqa/selftest/containerimage.py | 82 ++++++++++++++++++++++ 3 files changed, 98 insertions(+) create mode 100644 meta-selftest/recipes-test/container-image/container-image-testpkg.bb create mode 100644 meta-selftest/recipes-test/container-image/container-test-image.bb create mode 100644 meta/lib/oeqa/selftest/containerimage.py diff --git a/meta-selftest/recipes-test/container-image/container-image-testpkg.bb b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb new file mode 100644 index 0000000..f8dd229 --- /dev/null +++ b/meta-selftest/recipes-test/container-image/container-image-testpkg.bb @@ -0,0 +1,8 @@ +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +do_install_append() { + install -d ${D}${bindir} + touch ${D}${bindir}/theapp +} diff --git a/meta-selftest/recipes-test/container-image/container-test-image.bb b/meta-selftest/recipes-test/container-image/container-test-image.bb new file mode 100644 index 0000000..d5f939c --- /dev/null +++ b/meta-selftest/recipes-test/container-image/container-test-image.bb @@ -0,0 +1,8 @@ +IMAGE_INSTALL += "container-image-testpkg" + +LICENSE = "MIT" + +IMAGE_FSTYPES = "container" +IMAGE_LINGUAS = "" + +inherit core-image diff --git a/meta/lib/oeqa/selftest/containerimage.py b/meta/lib/oeqa/selftest/containerimage.py new file mode 100644 index 0000000..6a85222 --- /dev/null +++ b/meta/lib/oeqa/selftest/containerimage.py @@ -0,0 +1,82 @@ +import os + +from oeqa.selftest.base import oeSelfTest +from oeqa.utils.commands import bitbake, get_bb_vars, runCmd + +# This test builds an image with using the "container" IMAGE_FSTYPE, and +# ensures that then files in the image are only the ones expected. +# +# The only package added to the image is container_image_testpkg, which +# contains one file. However, due to some other things not cleaning up during +# rootfs creation, there is some cruft. Ideally bugs will be filed and the +# cruft removed, but for now we whitelist some known set. +# +# Also for performance reasons we're only checking the cruft when using ipk. +# When using deb, and rpm it is a bit different and we could test all +# of them, but this test is more to catch if other packages get added by +# default other than what is in ROOTFS_BOOTSTRAP_INSTALL. +# +class ContainerImageTests(oeSelfTest): + + # Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar" that + # the conversion type bar gets added as a dep as well + def test_expected_files(self): + + def get_each_path_part(path): + if path: + part = [ '.' + path + '/' ] + result = get_each_path_part(path.rsplit('/', 1)[0]) + if result: + return part + result + else: + return part + else: + return None + + bbvars = get_bb_vars(['bindir', 'sysconfdir', 'localstatedir', + 'DEPLOY_DIR_IMAGE']) + expected_files = [ + './', + '.{bindir}/theapp', + '.{sysconfdir}/default/', + '.{sysconfdir}/default/postinst', + '.{sysconfdir}/ld.so.cache', + '.{sysconfdir}/timestamp', + '.{sysconfdir}/version', + './run/', + '.{localstatedir}/cache/', + '.{localstatedir}/cache/ldconfig/', + '.{localstatedir}/cache/ldconfig/aux-cache', + '.{localstatedir}/cache/opkg/', + '.{localstatedir}/lib/', + '.{localstatedir}/lib/opkg/' + ] + + expected_files = [ x.format(bindir=bbvars['bindir'], + sysconfdir=bbvars['sysconfdir'], + localstatedir=bbvars['localstatedir']) + for x in expected_files ] + + # Since tar lists all directories individually, make sure each element + # from bindir, sysconfdir, etc is added + expected_files += get_each_path_part(bbvars['bindir']) + expected_files += get_each_path_part(bbvars['sysconfdir']) + expected_files += get_each_path_part(bbvars['localstatedir']) + + expected_files = sorted(expected_files) + + + self.write_config('PREFERRED_PROVIDER_virtual/kernel = "linux-dummy"') + self.append_config('MACHINE = "qemux86"') + self.append_config('IMAGE_FSTYPES = "container"') + self.append_config('PACKAGE_CLASSES = "package_ipk"') + + # Build the image of course + bitbake('container-test-image') + + image = os.path.join(bbvars['DEPLOY_DIR_IMAGE'], + 'container-test-image-qemux86.tar.bz2') + + # Ensure the files in the image are what we expect + result = runCmd("tar tf {} | sort".format(image), shell=True) + self.assertEqual(result.output.split('\n'), expected_files) -- 2.9.3 ^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-01-19 22:50 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-01-18 18:31 [PATCH 0/3] Add initial container IMAGE_FSTYPES support Randy Witt 2017-01-18 18:31 ` [PATCH 1/3] image-container.bbclass: Add the "container" IMAGE_FSTYPES Randy Witt 2017-01-18 18:31 ` [PATCH 2/3] image-container.bbclass: Error if not using linux-dummy Randy Witt 2017-01-18 18:31 ` [PATCH 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test Randy Witt 2017-01-19 12:01 ` Burton, Ross 2017-01-19 22:50 ` Randy Witt 2017-01-19 21:52 ` [PATCH] " Randy Witt
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox