* [RFC][PATCH] bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link
@ 2018-09-27 9:11 Martin Jansa
2018-09-27 9:34 ` ✗ patchtest: failure for " Patchwork
2018-09-27 11:15 ` [RFC][PATCH] " Richard Purdie
0 siblings, 2 replies; 4+ messages in thread
From: Martin Jansa @ 2018-09-27 9:11 UTC (permalink / raw)
To: openembedded-core
* just RFC, the part for images isn't finished yet and there is
still some issue with DATETIME when kernel artifacts are used
from sstate, this is just to validate the idea behind
[YOCTO #12937] before finishing the implementation (it's already
finished and used by various LGE builds, but having simpler
way of doing it directly in oe-core mighe be useful for others).
* move IMAGE_VERSION_SUFFIX from _NAME variables to _LINK_NAME
that way e.g. kernel.do_deploy can be reused from sstate to
provide "version-less" artifacts and then very fast
do_deploy_links task just adds links with consistent suffixes
* create hard links instead of symlinks, so that whatever version
the filename says is really there
* some IMAGE_FSTYPES might need the "version-less" IMAGE_NAME file
to be removed first or they might either append or update the
content of the image instead of creating new image file from
scratch - I have seen this only with one proprietary format we
generate with our own tool, so hopefully this isn't very common
* this is basically the mechanism are using in webOS with
WEBOS_IMAGE_NAME_SUFFIX which is for official builds set from
jenkins job and then all artifacts (images as well as corresponding
kernel files) have the same version string, the implementation
"from outside" is a bit tricky as shown in webOS OSE, the variable
modifications:
https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/conf/distro/include/webos.inc#L65
and then various bbclasses to hook do_webos_deploy_fixup task creating
the hardlinks for possible artifacts:
https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/webos_deploy.bbclass
https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/kernel.bbclass
https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e44a132d01297ff478248/meta-webos/classes/image.bbclass
so hopefully with all these changes in oe-core other project can
achieve the same just by setting one variable IMAGE_VERSION_SUFFIX
[YOCTO #12937]
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
meta/classes/kernel-artifact-names.bbclass | 4 +--
meta/classes/kernel.bbclass | 40 ++++++++++++++++------
meta/conf/bitbake.conf | 4 +--
3 files changed, 33 insertions(+), 15 deletions(-)
diff --git a/meta/classes/kernel-artifact-names.bbclass b/meta/classes/kernel-artifact-names.bbclass
index bbeecba7bd..84ec193b5a 100644
--- a/meta/classes/kernel-artifact-names.bbclass
+++ b/meta/classes/kernel-artifact-names.bbclass
@@ -1,5 +1,5 @@
-KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
-KERNEL_ARTIFACT_LINK_NAME ?= "${MACHINE}"
+KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-${MACHINE}"
+KERNEL_ARTIFACT_LINK_NAME ?= "${KERNEL_ARTIFACT_NAME}${IMAGE_VERSION_SUFFIX}"
KERNEL_IMAGE_NAME ?= "${KERNEL_ARTIFACT_NAME}"
KERNEL_IMAGE_LINK_NAME ?= "${KERNEL_ARTIFACT_LINK_NAME}"
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index d0fbbd1989..7aaebb56b4 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -667,25 +667,18 @@ kernel_do_deploy() {
fi
for imageType in ${KERNEL_IMAGETYPES} ; do
- base_name=${imageType}-${KERNEL_IMAGE_NAME}
- install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType} $deployDir/${base_name}.bin
- symlink_name=${imageType}-${KERNEL_IMAGE_LINK_NAME}
- ln -sf ${base_name}.bin $deployDir/${symlink_name}.bin
- ln -sf ${base_name}.bin $deployDir/${imageType}
+ install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType} $deployDir/${imageType}-${KERNEL_IMAGE_NAME}.bin
+ ln -sf ${imageType}-${KERNEL_IMAGE_NAME}.bin $deployDir/${imageType}
done
if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
mkdir -p ${D}${root_prefix}/lib
tar -cvzf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
- ln -sf modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
fi
if [ ! -z "${INITRAMFS_IMAGE}" -a x"${INITRAMFS_IMAGE_BUNDLE}" = x1 ]; then
for imageType in ${KERNEL_IMAGETYPES} ; do
- initramfs_base_name=${imageType}-${INITRAMFS_NAME}
- initramfs_symlink_name=${imageType}-${INITRAMFS_LINK_NAME}
- install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType}.initramfs $deployDir/${initramfs_base_name}.bin
- ln -sf ${initramfs_base_name}.bin $deployDir/${initramfs_symlink_name}.bin
+ install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType}.initramfs $deployDir/${imageType}-${INITRAMFS_NAME}.bin
done
fi
}
@@ -695,7 +688,32 @@ do_deploy[prefuncs] += "package_get_auto_pr"
addtask deploy after do_populate_sysroot do_packagedata
-EXPORT_FUNCTIONS do_deploy
+kernel_do_deploy_links() {
+ deployDir="${DEPLOY_DIR_IMAGE}"
+ if [ -n "${KERNEL_DEPLOYSUBDIR}" ]; then
+ deployDir="${DEPLOY_DIR_IMAGE}/${KERNEL_DEPLOYSUBDIR}"
+ mkdir "$deployDir"
+ fi
+
+ for imageType in ${KERNEL_IMAGETYPES} ; do
+ ln -vf $deployDir/${imageType}-${KERNEL_IMAGE_NAME}.bin $deployDir/${imageType}-${KERNEL_IMAGE_LINK_NAME}.bin
+ done
+
+ if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
+ ln -vf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
+ fi
+
+ if [ ! -z "${INITRAMFS_IMAGE}" -a "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
+ for imageType in ${KERNEL_IMAGETYPES} ; do
+ ln -vf ${imageType}-${INITRAMFS_NAME}.bin ${imageType}-${INITRAMFS_LINK_NAME}.bin
+ done
+ fi
+}
+do_deploy_links[prefuncs] += "package_get_auto_pr"
+
+addtask deploy_links after do_deploy before do_build
+
+EXPORT_FUNCTIONS do_deploy do_deploy_links
# Add using Device Tree support
inherit kernel-devicetree
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 93aee1ae46..017cb163ba 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -447,8 +447,8 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"
IMAGE_BASENAME = "${PN}"
IMAGE_VERSION_SUFFIX = "-${DATETIME}"
IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
-IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
-IMAGE_LINK_NAME = "${IMAGE_BASENAME}-${MACHINE}"
+IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}"
+IMAGE_LINK_NAME = "${IMAGE_NAME}${IMAGE_VERSION_SUFFIX}"
# This option allows for a percentage overage of the actual image size rather than a
# fixed extra space, this is space needed for initial startup and basic operations.
--
2.17.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* ✗ patchtest: failure for bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link
2018-09-27 9:11 [RFC][PATCH] bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link Martin Jansa
@ 2018-09-27 9:34 ` Patchwork
2018-09-27 11:15 ` [RFC][PATCH] " Richard Purdie
1 sibling, 0 replies; 4+ messages in thread
From: Patchwork @ 2018-09-27 9:34 UTC (permalink / raw)
To: Martin Jansa; +Cc: openembedded-core
== Series Details ==
Series: bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link
Revision: 1
URL : https://patchwork.openembedded.org/series/14258/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:
* Patch [RFC] bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link
Issue Commit shortlog is too long [test_shortlog_length]
Suggested fix Edit shortlog so that it is 90 characters or less (currently 119 characters)
If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC][PATCH] bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link
2018-09-27 9:11 [RFC][PATCH] bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link Martin Jansa
2018-09-27 9:34 ` ✗ patchtest: failure for " Patchwork
@ 2018-09-27 11:15 ` Richard Purdie
2018-09-27 12:13 ` Martin Jansa
1 sibling, 1 reply; 4+ messages in thread
From: Richard Purdie @ 2018-09-27 11:15 UTC (permalink / raw)
To: Martin Jansa, openembedded-core
Hi Martin,
In the commit message you say a lot about what you've changed but not
so much about why the changes are important and the advantages they
bring.
There are tradeoffs, for example symlinks make it clear which artefact
they're really pointing at, hardlinks hide that fact, you need to go
and look at inode numbers to figure out which of several artefacts one
might be pointing at. I'm not sure that is an improvement.
I'm also slightly concerned that we need to bypass sstate control, the
whole intent there was to ensure that output is reproducible and
consistently restored. Adding in a new task means do_build target usage
will work but any of the code that has do_deploy as a dependency (e.g.
an recrdep) will now also need to consider do_deploy_links. For that
reason alone, I'm not sure we can do this.
Cheers,
Richard
On Thu, 2018-09-27 at 09:11 +0000, Martin Jansa wrote:
> * just RFC, the part for images isn't finished yet and there is
> still some issue with DATETIME when kernel artifacts are used
> from sstate, this is just to validate the idea behind
> [YOCTO #12937] before finishing the implementation (it's already
> finished and used by various LGE builds, but having simpler
> way of doing it directly in oe-core mighe be useful for others).
>
> * move IMAGE_VERSION_SUFFIX from _NAME variables to _LINK_NAME
> that way e.g. kernel.do_deploy can be reused from sstate to
> provide "version-less" artifacts and then very fast
> do_deploy_links task just adds links with consistent suffixes
> * create hard links instead of symlinks, so that whatever version
> the filename says is really there
> * some IMAGE_FSTYPES might need the "version-less" IMAGE_NAME file
> to be removed first or they might either append or update the
> content of the image instead of creating new image file from
> scratch - I have seen this only with one proprietary format we
> generate with our own tool, so hopefully this isn't very common
> * this is basically the mechanism are using in webOS with
> WEBOS_IMAGE_NAME_SUFFIX which is for official builds set from
> jenkins job and then all artifacts (images as well as corresponding
> kernel files) have the same version string, the implementation
> "from outside" is a bit tricky as shown in webOS OSE, the variable
> modifications:
> https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> 44a132d01297ff478248/meta-webos/conf/distro/include/webos.inc#L65
> and then various bbclasses to hook do_webos_deploy_fixup task
> creating
> the hardlinks for possible artifacts:
> https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> 44a132d01297ff478248/meta-webos/classes/webos_deploy.bbclass
> https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> 44a132d01297ff478248/meta-webos/classes/kernel.bbclass
> https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> 44a132d01297ff478248/meta-webos/classes/image.bbclass
> so hopefully with all these changes in oe-core other project can
> achieve the same just by setting one variable IMAGE_VERSION_SUFFIX
>
> [YOCTO #12937]
>
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
> meta/classes/kernel-artifact-names.bbclass | 4 +--
> meta/classes/kernel.bbclass | 40 ++++++++++++++++--
> ----
> meta/conf/bitbake.conf | 4 +--
> 3 files changed, 33 insertions(+), 15 deletions(-)
>
> diff --git a/meta/classes/kernel-artifact-names.bbclass
> b/meta/classes/kernel-artifact-names.bbclass
> index bbeecba7bd..84ec193b5a 100644
> --- a/meta/classes/kernel-artifact-names.bbclass
> +++ b/meta/classes/kernel-artifact-names.bbclass
> @@ -1,5 +1,5 @@
> -KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-
> ${MACHINE}${IMAGE_VERSION_SUFFIX}"
> -KERNEL_ARTIFACT_LINK_NAME ?= "${MACHINE}"
> +KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-${MACHINE}"
> +KERNEL_ARTIFACT_LINK_NAME ?=
> "${KERNEL_ARTIFACT_NAME}${IMAGE_VERSION_SUFFIX}"
>
> KERNEL_IMAGE_NAME ?= "${KERNEL_ARTIFACT_NAME}"
> KERNEL_IMAGE_LINK_NAME ?= "${KERNEL_ARTIFACT_LINK_NAME}"
> diff --git a/meta/classes/kernel.bbclass
> b/meta/classes/kernel.bbclass
> index d0fbbd1989..7aaebb56b4 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -667,25 +667,18 @@ kernel_do_deploy() {
> fi
>
> for imageType in ${KERNEL_IMAGETYPES} ; do
> - base_name=${imageType}-${KERNEL_IMAGE_NAME}
> - install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType}
> $deployDir/${base_name}.bin
> - symlink_name=${imageType}-${KERNEL_IMAGE_LINK_NAME}
> - ln -sf ${base_name}.bin
> $deployDir/${symlink_name}.bin
> - ln -sf ${base_name}.bin $deployDir/${imageType}
> + install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType}
> $deployDir/${imageType}-${KERNEL_IMAGE_NAME}.bin
> + ln -sf ${imageType}-${KERNEL_IMAGE_NAME}.bin
> $deployDir/${imageType}
> done
>
> if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e
> '^CONFIG_MODULES=y$' .config); then
> mkdir -p ${D}${root_prefix}/lib
> tar -cvzf $deployDir/modules-
> ${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
> - ln -sf modules-${MODULE_TARBALL_NAME}.tgz
> $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
> fi
>
> if [ ! -z "${INITRAMFS_IMAGE}" -a
> x"${INITRAMFS_IMAGE_BUNDLE}" = x1 ]; then
> for imageType in ${KERNEL_IMAGETYPES} ; do
> - initramfs_base_name=${imageType}-
> ${INITRAMFS_NAME}
> - initramfs_symlink_name=${imageType}-
> ${INITRAMFS_LINK_NAME}
> - install -m 0644
> ${KERNEL_OUTPUT_DIR}/${imageType}.initramfs
> $deployDir/${initramfs_base_name}.bin
> - ln -sf ${initramfs_base_name}.bin
> $deployDir/${initramfs_symlink_name}.bin
> + install -m 0644
> ${KERNEL_OUTPUT_DIR}/${imageType}.initramfs $deployDir/${imageType}-
> ${INITRAMFS_NAME}.bin
> done
> fi
> }
> @@ -695,7 +688,32 @@ do_deploy[prefuncs] += "package_get_auto_pr"
>
> addtask deploy after do_populate_sysroot do_packagedata
>
> -EXPORT_FUNCTIONS do_deploy
> +kernel_do_deploy_links() {
> + deployDir="${DEPLOY_DIR_IMAGE}"
> + if [ -n "${KERNEL_DEPLOYSUBDIR}" ]; then
> + deployDir="${DEPLOY_DIR_IMAGE}/${KERNEL_DEPLOYSUBDIR
> }"
> + mkdir "$deployDir"
> + fi
> +
> + for imageType in ${KERNEL_IMAGETYPES} ; do
> + ln -vf $deployDir/${imageType}-
> ${KERNEL_IMAGE_NAME}.bin $deployDir/${imageType}-
> ${KERNEL_IMAGE_LINK_NAME}.bin
> + done
> +
> + if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e
> '^CONFIG_MODULES=y$' .config); then
> + ln -vf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
> $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
> + fi
> +
> + if [ ! -z "${INITRAMFS_IMAGE}" -a
> "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
> + for imageType in ${KERNEL_IMAGETYPES} ; do
> + ln -vf ${imageType}-${INITRAMFS_NAME}.bin
> ${imageType}-${INITRAMFS_LINK_NAME}.bin
> + done
> + fi
> +}
> +do_deploy_links[prefuncs] += "package_get_auto_pr"
> +
> +addtask deploy_links after do_deploy before do_build
> +
> +EXPORT_FUNCTIONS do_deploy do_deploy_links
>
> # Add using Device Tree support
> inherit kernel-devicetree
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 93aee1ae46..017cb163ba 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -447,8 +447,8 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"
> IMAGE_BASENAME = "${PN}"
> IMAGE_VERSION_SUFFIX = "-${DATETIME}"
> IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
> -IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
> -IMAGE_LINK_NAME = "${IMAGE_BASENAME}-${MACHINE}"
> +IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}"
> +IMAGE_LINK_NAME = "${IMAGE_NAME}${IMAGE_VERSION_SUFFIX}"
>
> # This option allows for a percentage overage of the actual image
> size rather than a
> # fixed extra space, this is space needed for initial startup and
> basic operations.
> --
> 2.17.1
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFC][PATCH] bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link
2018-09-27 11:15 ` [RFC][PATCH] " Richard Purdie
@ 2018-09-27 12:13 ` Martin Jansa
0 siblings, 0 replies; 4+ messages in thread
From: Martin Jansa @ 2018-09-27 12:13 UTC (permalink / raw)
To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 11808 bytes --]
The reason is that for our builds we set the IMAGE_VERSION_SUFFIX based on
jenkins job number which creates them (or you can
imagine IMAGE_VERSION_SUFFIX being set to something like "release-2.6").
With current metadata we have 2 options:
1) keep IMAGE_VERSION_SUFFIX as is, but don't vardepexclude the build
number (DATETIME in the default value)
But this causes e.g. kernel.do_deploy signature to be changed and whole
do_deploy task re-executed (instead of reused from sstate if it's the same
except the output files names), which in turn causes whole kernel to be
rebuilt in every build, because to run do_deploy we need do_install etc
(and these builds are usually using rm_work and TMPDIR is empty before each
job).
2) leave IMAGE_VERSION_SUFFIX as is, but rename the artifacts after the
build
When the image artifact says "image-release-2.6.img" and kernel image from
the same build says "uImage-release-2.5.bin" (because kernel wasn't
modified at all since previous release), people will be confused.
This is what we were seeing in CI which was usually sending just the image,
but then for some new MACHINEs we needed to send kernel artifact separately
and sometimes the artifact wasn't found with expected name.
Then we changed the build number (aka IMAGE_VERSION_SUFFIX) to be included
in do_deploy signatures which lead to 1) and kernel rebuilds every time
To solve both, we've added webos_deploy_fixup tasks (in many places) which
just adds the hardlinks with expected filenames (so that other systems just
using the artifacts doesn't need to know anything about internal issues
with sstate reuse, nor needing external script to rename things after the
build).
Hardlinks are better than symlinks in this case, because you don't want to
have symlink
image-release-2.5.img -> image.img
still in the deploy directory after you do another build which creates new
image.img and new symlink
image-release-2.6.img -> image.img
If someone downloads image-release-2.5.img "too late" he will get 2.6
version instead.
Hardlinks don't have this issue and yes the need to check the inode numbers
might be confusing to some, but when do you really care where does it
point? In our use-case we always care only about the *"release-2.5"* and
the artifacts without the version string are considered as intermediate
files created by the build or "whatever version was built last in this
build directory", pretty much the same as version-less symlinks in current
build (which are easy to resolve when looking at the filesystem, but
exposing those symlinks for download e.g. over HTTP hides where it points
completely.
I'm not sure I understand your concern about bypassing sstate control, we
still want to reuse kernel.do_deploy and image.do_image_complete. Just with
additional very fast intentionally not-covered-by-sstate task before
do_build to create consistent names.
On Thu, Sep 27, 2018 at 1:15 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> Hi Martin,
>
> In the commit message you say a lot about what you've changed but not
> so much about why the changes are important and the advantages they
> bring.
>
> There are tradeoffs, for example symlinks make it clear which artefact
> they're really pointing at, hardlinks hide that fact, you need to go
> and look at inode numbers to figure out which of several artefacts one
> might be pointing at. I'm not sure that is an improvement.
>
> I'm also slightly concerned that we need to bypass sstate control, the
> whole intent there was to ensure that output is reproducible and
> consistently restored. Adding in a new task means do_build target usage
> will work but any of the code that has do_deploy as a dependency (e.g.
> an recrdep) will now also need to consider do_deploy_links. For that
> reason alone, I'm not sure we can do this.
>
> Cheers,
>
> Richard
>
> On Thu, 2018-09-27 at 09:11 +0000, Martin Jansa wrote:
> > * just RFC, the part for images isn't finished yet and there is
> > still some issue with DATETIME when kernel artifacts are used
> > from sstate, this is just to validate the idea behind
> > [YOCTO #12937] before finishing the implementation (it's already
> > finished and used by various LGE builds, but having simpler
> > way of doing it directly in oe-core mighe be useful for others).
> >
> > * move IMAGE_VERSION_SUFFIX from _NAME variables to _LINK_NAME
> > that way e.g. kernel.do_deploy can be reused from sstate to
> > provide "version-less" artifacts and then very fast
> > do_deploy_links task just adds links with consistent suffixes
> > * create hard links instead of symlinks, so that whatever version
> > the filename says is really there
> > * some IMAGE_FSTYPES might need the "version-less" IMAGE_NAME file
> > to be removed first or they might either append or update the
> > content of the image instead of creating new image file from
> > scratch - I have seen this only with one proprietary format we
> > generate with our own tool, so hopefully this isn't very common
> > * this is basically the mechanism are using in webOS with
> > WEBOS_IMAGE_NAME_SUFFIX which is for official builds set from
> > jenkins job and then all artifacts (images as well as corresponding
> > kernel files) have the same version string, the implementation
> > "from outside" is a bit tricky as shown in webOS OSE, the variable
> > modifications:
> > https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> > 44a132d01297ff478248/meta-webos/conf/distro/include/webos.inc#L65
> > and then various bbclasses to hook do_webos_deploy_fixup task
> > creating
> > the hardlinks for possible artifacts:
> > https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> > 44a132d01297ff478248/meta-webos/classes/webos_deploy.bbclass
> > https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> > 44a132d01297ff478248/meta-webos/classes/kernel.bbclass
> > https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> > 44a132d01297ff478248/meta-webos/classes/image.bbclass
> > so hopefully with all these changes in oe-core other project can
> > achieve the same just by setting one variable IMAGE_VERSION_SUFFIX
> >
> > [YOCTO #12937]
> >
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> > meta/classes/kernel-artifact-names.bbclass | 4 +--
> > meta/classes/kernel.bbclass | 40 ++++++++++++++++--
> > ----
> > meta/conf/bitbake.conf | 4 +--
> > 3 files changed, 33 insertions(+), 15 deletions(-)
> >
> > diff --git a/meta/classes/kernel-artifact-names.bbclass
> > b/meta/classes/kernel-artifact-names.bbclass
> > index bbeecba7bd..84ec193b5a 100644
> > --- a/meta/classes/kernel-artifact-names.bbclass
> > +++ b/meta/classes/kernel-artifact-names.bbclass
> > @@ -1,5 +1,5 @@
> > -KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-
> > ${MACHINE}${IMAGE_VERSION_SUFFIX}"
> > -KERNEL_ARTIFACT_LINK_NAME ?= "${MACHINE}"
> > +KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-${MACHINE}"
> > +KERNEL_ARTIFACT_LINK_NAME ?=
> > "${KERNEL_ARTIFACT_NAME}${IMAGE_VERSION_SUFFIX}"
> >
> > KERNEL_IMAGE_NAME ?= "${KERNEL_ARTIFACT_NAME}"
> > KERNEL_IMAGE_LINK_NAME ?= "${KERNEL_ARTIFACT_LINK_NAME}"
> > diff --git a/meta/classes/kernel.bbclass
> > b/meta/classes/kernel.bbclass
> > index d0fbbd1989..7aaebb56b4 100644
> > --- a/meta/classes/kernel.bbclass
> > +++ b/meta/classes/kernel.bbclass
> > @@ -667,25 +667,18 @@ kernel_do_deploy() {
> > fi
> >
> > for imageType in ${KERNEL_IMAGETYPES} ; do
> > - base_name=${imageType}-${KERNEL_IMAGE_NAME}
> > - install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType}
> > $deployDir/${base_name}.bin
> > - symlink_name=${imageType}-${KERNEL_IMAGE_LINK_NAME}
> > - ln -sf ${base_name}.bin
> > $deployDir/${symlink_name}.bin
> > - ln -sf ${base_name}.bin $deployDir/${imageType}
> > + install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType}
> > $deployDir/${imageType}-${KERNEL_IMAGE_NAME}.bin
> > + ln -sf ${imageType}-${KERNEL_IMAGE_NAME}.bin
> > $deployDir/${imageType}
> > done
> >
> > if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e
> > '^CONFIG_MODULES=y$' .config); then
> > mkdir -p ${D}${root_prefix}/lib
> > tar -cvzf $deployDir/modules-
> > ${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
> > - ln -sf modules-${MODULE_TARBALL_NAME}.tgz
> > $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
> > fi
> >
> > if [ ! -z "${INITRAMFS_IMAGE}" -a
> > x"${INITRAMFS_IMAGE_BUNDLE}" = x1 ]; then
> > for imageType in ${KERNEL_IMAGETYPES} ; do
> > - initramfs_base_name=${imageType}-
> > ${INITRAMFS_NAME}
> > - initramfs_symlink_name=${imageType}-
> > ${INITRAMFS_LINK_NAME}
> > - install -m 0644
> > ${KERNEL_OUTPUT_DIR}/${imageType}.initramfs
> > $deployDir/${initramfs_base_name}.bin
> > - ln -sf ${initramfs_base_name}.bin
> > $deployDir/${initramfs_symlink_name}.bin
> > + install -m 0644
> > ${KERNEL_OUTPUT_DIR}/${imageType}.initramfs $deployDir/${imageType}-
> > ${INITRAMFS_NAME}.bin
> > done
> > fi
> > }
> > @@ -695,7 +688,32 @@ do_deploy[prefuncs] += "package_get_auto_pr"
> >
> > addtask deploy after do_populate_sysroot do_packagedata
> >
> > -EXPORT_FUNCTIONS do_deploy
> > +kernel_do_deploy_links() {
> > + deployDir="${DEPLOY_DIR_IMAGE}"
> > + if [ -n "${KERNEL_DEPLOYSUBDIR}" ]; then
> > + deployDir="${DEPLOY_DIR_IMAGE}/${KERNEL_DEPLOYSUBDIR
> > }"
> > + mkdir "$deployDir"
> > + fi
> > +
> > + for imageType in ${KERNEL_IMAGETYPES} ; do
> > + ln -vf $deployDir/${imageType}-
> > ${KERNEL_IMAGE_NAME}.bin $deployDir/${imageType}-
> > ${KERNEL_IMAGE_LINK_NAME}.bin
> > + done
> > +
> > + if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e
> > '^CONFIG_MODULES=y$' .config); then
> > + ln -vf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
> > $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
> > + fi
> > +
> > + if [ ! -z "${INITRAMFS_IMAGE}" -a
> > "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
> > + for imageType in ${KERNEL_IMAGETYPES} ; do
> > + ln -vf ${imageType}-${INITRAMFS_NAME}.bin
> > ${imageType}-${INITRAMFS_LINK_NAME}.bin
> > + done
> > + fi
> > +}
> > +do_deploy_links[prefuncs] += "package_get_auto_pr"
> > +
> > +addtask deploy_links after do_deploy before do_build
> > +
> > +EXPORT_FUNCTIONS do_deploy do_deploy_links
> >
> > # Add using Device Tree support
> > inherit kernel-devicetree
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 93aee1ae46..017cb163ba 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -447,8 +447,8 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"
> > IMAGE_BASENAME = "${PN}"
> > IMAGE_VERSION_SUFFIX = "-${DATETIME}"
> > IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
> > -IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
> > -IMAGE_LINK_NAME = "${IMAGE_BASENAME}-${MACHINE}"
> > +IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}"
> > +IMAGE_LINK_NAME = "${IMAGE_NAME}${IMAGE_VERSION_SUFFIX}"
> >
> > # This option allows for a percentage overage of the actual image
> > size rather than a
> > # fixed extra space, this is space needed for initial startup and
> > basic operations.
> > --
> > 2.17.1
> >
>
[-- Attachment #2: Type: text/html, Size: 14625 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2018-09-27 12:13 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-09-27 9:11 [RFC][PATCH] bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link Martin Jansa
2018-09-27 9:34 ` ✗ patchtest: failure for " Patchwork
2018-09-27 11:15 ` [RFC][PATCH] " Richard Purdie
2018-09-27 12:13 ` Martin Jansa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox