[PATCH 0/7] kernel: consolidated pull for developer experience

[PATCH 0/7] kernel: consolidated pull for developer experience

[Bruce Ashfield]

Richard,

Here are the first (and largest) kernel developer experience changes for the 1.8
release.

Obviously you are the author of patch 1 in the series, and I've made changes to
fixup the remaining out of tree module build, and on-target kernel build issues.

With these changes in place, I've built and booted the linux-yocto kernels, as
well as build the kernel itself on the target .. so they are definitely sane
from that point of view. I won't swear that there aren't any cornder cases still
lurking, but we need more eyes and tests to find those cases.

With this series, we've move a lot of the source copying and I/O intensive
operations to the kernelsrc recipe .. which makes them optional and not an
impact on the typical build/developer. As described in pach 1, the kernel is
unpacked, manipulated and built directly from STAGING_DIR_KERNEL.

If the src is required, the kerneldevsrc recipe can be added to the
DEPENDS/RDEPENDS and the packaging will happen (with the associated overhead).

I've kept the series history as you see it on purpose, so we can capture
credit for the optimization idea, and to make the changes more digestible
to those reading the series. If we want some additional squashing of the
commits, I'm happy to do that as well.

The final parts of this series are some collected kernel build fixes that 
I've had for a while, but are minor in the grand scheme of things.

Cheers,

Bruce

Bruce Ashfield (5):
  kernel: fix out of tree module builds
  kerneldev: create kernel-devsrc packaging
  images: introduce core-image-kernel-dev
  lttng/perf: depend on virtual/kernel:do_install
  kernel-yocto: fix non-git builds

Jeff Wang (1):
  kernel-yocto: make sure git tags get dereferenced properly in
    do_patch()

Richard Purdie (1):
  kernel: Rearrange for 1.8

 meta/classes/kernel-yocto.bbclass                  |  8 +-
 meta/classes/kernel.bbclass                        | 93 ++++------------------
 meta/classes/linux-kernel-base.bbclass             | 13 ++-
 .../images/core-image-kernel-dev.bb                | 17 ++++
 meta/recipes-kernel/linux/kernel-devsrc.bb         | 50 ++++++++++++
 meta/recipes-kernel/lttng/lttng-modules_2.5.0.bb   |  1 +
 meta/recipes-kernel/perf/perf.bb                   | 12 +--
 7 files changed, 104 insertions(+), 90 deletions(-)
 create mode 100644 meta/recipes-extended/images/core-image-kernel-dev.bb
 create mode 100644 meta/recipes-kernel/linux/kernel-devsrc.bb

-- 
2.1.0

[PATCH 1/7] kernel: Rearrange for 1.8

[Bruce Ashfield]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

In 1.8 we want to streamline the kernel build process. Basically we
currently have multiple copies of the kernel source floating around
and the copying/compression/decompression is painful.

Lets assume we have a kernel source per machine since in most cases
this is true (and we have a sysroot per machine anyway). Basically,
instead of extracting a source into WORKDIR, then copying to a sysroot,
we now set S to point straight at STAGING_DIR_KERNEL.

Anything using kernel source can then just point at it and use:

do_configure[depends] += "virtual/kernel:do_patch"

to depend on the kernel source being present. Note this is different
behaviour to DEPENDS += "virtual/kernel" which equates to
do_configure[depends] += "virtual/kernel:do_populate_sysroot".

Once we do this, we no longer need the copy operation in
do_populate_sysroot, in fact there is nothing to do there (yay).

The remaining part of the challenge is to kill off the horrible
do_install. This patch splits it off to a different class, the idea here
is to have a separate recipe which depends on the virtual/kernel:do_patch
and just installs and packages the source needed to build modules on
target into a specific package.

Right now this code is proof of concept. It builds kernels and kernel
modules. perf blows up in do_package with issues on finding the kernel
version which can probably be fixed by adding back the right bit of do_install,
and adding a dependency of do_package[depends] += "virtual/kernel:do_install"
to perf. The whole thing needs a good write up, the corner cases testing
and probably a good dose of cleanup to the remaining code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/kernel-yocto.bbclass      |  2 -
 meta/classes/kernel.bbclass            | 91 ++--------------------------------
 meta/classes/linux-kernel-base.bbclass | 13 ++++-
 meta/recipes-kernel/perf/perf.bb       | 12 ++---
 4 files changed, 20 insertions(+), 98 deletions(-)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index c3e24943bbaf..45bdc9670afe 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -1,5 +1,3 @@
-S = "${WORKDIR}/linux"
-
 # remove tasks that modify the source tree in case externalsrc is inherited
 SRCTREECOVEREDTASKS += "do_kernel_link_vmlinux do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_patch"
 
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index fa06c3aae6f3..0d335a589121 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -3,6 +3,8 @@ inherit linux-kernel-base kernel-module-split
 PROVIDES += "virtual/kernel"
 DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native depmodwrapper-cross bc-native"
 
+S = "${STAGING_DIR_TARGET}/${KERNEL_SRC_PATH}"
+
 # we include gcc above, we dont need virtual/libc
 INHIBIT_DEFAULT_DEPS = "1"
 
@@ -55,7 +57,7 @@ KERNEL_IMAGEDEST = "boot"
 #
 export CMDLINE_CONSOLE = "console=${@d.getVar("KERNEL_CONSOLE",1) or "ttyS0"}"
 
-KERNEL_VERSION = "${@get_kernelversion('${B}')}"
+KERNEL_VERSION = "${@get_kernelversion_headers('${B}')}"
 
 KERNEL_LOCALVERSION ?= ""
 
@@ -221,93 +223,6 @@ kernel_do_install() {
 	#
 
 	echo "${KERNEL_VERSION}" > $kerneldir/kernel-abiversion
-
-	#
-	# Store kernel image name to allow use during image generation
-	#
-
-	echo "${KERNEL_IMAGE_BASE_NAME}" >$kerneldir/kernel-image-name
-
-	#
-	# Copy the entire source tree. In case an external build directory is
-	# used, copy the build directory over first, then copy over the source
-	# dir. This ensures the original Makefiles are used and not the
-	# redirecting Makefiles in the build directory.
-	#
-	find . -depth -not -name "*.cmd" -not -name "*.o" -not -name "*.so.dbg" -not -name "*.so" -not -path "./Documentation*" -not -path "./source*" -not -path "./.*" -print0 | cpio --null -pdlu $kerneldir
-	cp .config $kerneldir
-	if [ "${S}" != "${B}" ]; then
-		pwd="$PWD"
-		cd "${S}"
-		find . -depth -not -path "./Documentation*" -not -path "./.*" -print0 | cpio --null -pdlu $kerneldir
-		cd "$pwd"
-	fi
-
-	# Test to ensure that the output file and image type are not actually
-	# the same file. If hardlinking is used, they will be the same, and there's
-	# no need to install.
-	! [ ${KERNEL_OUTPUT} -ef $kerneldir/${KERNEL_IMAGETYPE} ] && install -m 0644 ${KERNEL_OUTPUT} $kerneldir/${KERNEL_IMAGETYPE}
-	install -m 0644 System.map $kerneldir/System.map-${KERNEL_VERSION}
-
-	# Dummy Makefile so the clean below works
-        mkdir $kerneldir/Documentation
-        touch $kerneldir/Documentation/Makefile
-
-	#
-	# Clean and remove files not needed for building modules.
-	# Some distributions go through a lot more trouble to strip out
-	# unecessary headers, for now, we just prune the obvious bits.
-	#
-	# We don't want to leave host-arch binaries in /sysroots, so
-	# we clean the scripts dir while leaving the generated config
-	# and include files.
-	#
-	oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts 
-
-	# hide directories that shouldn't have their .c, s and S files deleted
-	for d in tools scripts lib; do
-		mv $kerneldir/$d $kerneldir/.$d
-	done
-
-	# delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is 
-	# much faster than find -prune and -exec
-	find $kerneldir -not -path '*/\.*' -type f -name "*.[csS]" -delete
-
-	# put the hidden dirs back
-	for d in tools scripts lib; do
-		mv $kerneldir/.$d $kerneldir/$d
-	done
-
-	# As of Linux kernel version 3.0.1, the clean target removes
-	# arch/powerpc/lib/crtsavres.o which is present in
-	# KBUILD_LDFLAGS_MODULE, making it required to build external modules.
-	if [ ${ARCH} = "powerpc" ]; then
-		cp -l arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o
-	fi
-
-	# Necessary for building modules like compat-wireless.
-	if [ -f include/generated/bounds.h ]; then
-		cp -l include/generated/bounds.h $kerneldir/include/generated/bounds.h
-	fi
-	if [ -d arch/${ARCH}/include/generated ]; then
-		mkdir -p $kerneldir/arch/${ARCH}/include/generated/
-		cp -flR arch/${ARCH}/include/generated/* $kerneldir/arch/${ARCH}/include/generated/
-	fi
-
-	# Remove the following binaries which cause strip or arch QA errors
-	# during do_package for cross-compiled platforms
-	bin_files="arch/powerpc/boot/addnote arch/powerpc/boot/hack-coff \
-	           arch/powerpc/boot/mktree scripts/kconfig/zconf.tab.o \
-		   scripts/kconfig/conf.o scripts/kconfig/kxgettext.o"
-	for entry in $bin_files; do
-		rm -f $kerneldir/$entry
-	done
-
-	# kernels <2.6.30 don't have $kerneldir/tools directory so we check if it exists before calling sed
-	if [ -f $kerneldir/tools/perf/Makefile ]; then
-		# Fix SLANG_INC for slang.h
-		sed -i 's#-I/usr/include/slang#-I=/usr/include/slang#g' $kerneldir/tools/perf/Makefile
-	fi
 }
 do_install[prefuncs] += "package_get_auto_pr"
 
diff --git a/meta/classes/linux-kernel-base.bbclass b/meta/classes/linux-kernel-base.bbclass
index 4f2b0a4a9844..89ce71605c00 100644
--- a/meta/classes/linux-kernel-base.bbclass
+++ b/meta/classes/linux-kernel-base.bbclass
@@ -1,5 +1,5 @@
 # parse kernel ABI version out of <linux/version.h>
-def get_kernelversion(p):
+def get_kernelversion_headers(p):
     import re
 
     fn = p + '/include/linux/utsrelease.h'
@@ -9,7 +9,6 @@ def get_kernelversion(p):
     if not os.path.isfile(fn):
         fn = p + '/include/linux/version.h'
 
-    import re
     try:
         f = open(fn, 'r')
     except IOError:
@@ -24,6 +23,16 @@ def get_kernelversion(p):
             return m.group(1)
     return None
 
+
+def get_kernelversion_file(p):
+    fn = p + '/kernel-abiversion'
+
+    try:
+        with open(fn, 'r') as f:
+            return f.readlines()[0].strip()
+    except IOError:
+        return None
+
 def linux_module_packages(s, d):
 	suffix = ""
 	return " ".join(map(lambda s: "kernel-module-%s%s" % (s.lower().replace('_', '-').replace('@', '+'), suffix), s.split()))
diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
index 3771cdb3e9dd..d375e246ce30 100644
--- a/meta/recipes-kernel/perf/perf.bb
+++ b/meta/recipes-kernel/perf/perf.bb
@@ -22,7 +22,7 @@ TUI_DEPENDS = "${@perf_feature_enabled('perf-tui', 'libnewt slang', '',d)}"
 SCRIPTING_DEPENDS = "${@perf_feature_enabled('perf-scripting', 'perl python', '',d)}"
 LIBUNWIND_DEPENDS = "${@perf_feature_enabled('perf-libunwind', 'libunwind', '',d)}"
 
-DEPENDS = "virtual/kernel \
+DEPENDS = " \
     virtual/${MLPREFIX}libc \
     ${MLPREFIX}elfutils \
     ${MLPREFIX}binutils \
@@ -32,6 +32,8 @@ DEPENDS = "virtual/kernel \
     bison flex \
 "
 
+do_configure[depends] += "virtual/kernel:do_patch"
+
 PROVIDES = "virtual/perf"
 
 inherit linux-kernel-base kernel-arch pythonnative
@@ -46,7 +48,7 @@ export HOST_SYS
 #kernel 3.1+ supports WERROR to disable warnings as errors
 export WERROR = "0"
 
-do_populate_lic[depends] += "virtual/kernel:do_populate_sysroot"
+do_populate_lic[depends] += "virtual/kernel:do_patch"
 
 # needed for building the tools/perf Perl binding
 inherit perlnative cpan-base
@@ -56,9 +58,7 @@ export PERL_INC = "${STAGING_LIBDIR}${PERL_OWN_DIR}/perl/${@get_perl_version(d)}
 export PERL_LIB = "${STAGING_LIBDIR}${PERL_OWN_DIR}/perl/${@get_perl_version(d)}"
 export PERL_ARCHLIB = "${STAGING_LIBDIR}${PERL_OWN_DIR}/perl/${@get_perl_version(d)}"
 
-S = "${STAGING_KERNEL_DIR}"
-# The source should be ready after the do_unpack
-do_unpack[depends] += "virtual/kernel:do_populate_sysroot"
+inherit kernelsrc
 
 B = "${WORKDIR}/${BPN}-${PV}"
 
@@ -157,7 +157,7 @@ do_configure_prepend () {
 }
 
 python do_package_prepend() {
-    bb.data.setVar('PKGV', get_kernelversion('${S}').split("-")[0], d)
+    d.setVar('PKGV', d.getVar("KERNEL_VERSION", True).split("-")[0])
 }
 
 PACKAGE_ARCH = "${MACHINE_ARCH}"
-- 
2.1.0

[PATCH 2/7] kernel: fix out of tree module builds

[Bruce Ashfield]

With the kernel build optimizations, we no longer copy the source from
the built kernel into the staging dir, since the kernel is unpacked and built
directly from the staging dir.

This means that a few build artifacts need to be restored to allow out of
tree modules to once again build.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/classes/kernel.bbclass | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 0d335a589121..348041449cb0 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -223,6 +223,28 @@ kernel_do_install() {
 	#
 
 	echo "${KERNEL_VERSION}" > $kerneldir/kernel-abiversion
+	
+	# Copy files required for module builds
+	cp System.map $kerneldir/System.map-${KERNEL_VERSION}
+	cp Module.symvers $kerneldir/
+	cp .config $kerneldir/
+	mkdir -p $kerneldir/include/config
+	cp include/config/kernel.release $kerneldir/include/config/kernel.release
+
+	# As of Linux kernel version 3.0.1, the clean target removes
+	# arch/powerpc/lib/crtsavres.o which is present in
+	# KBUILD_LDFLAGS_MODULE, making it required to build external modules.
+	if [ ${ARCH} = "powerpc" ]; then
+		cp arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o
+	fi
+
+	mkdir -p $kerneldir/include/generated/
+	cp -fR include/generated/* $kerneldir/include/generated/
+
+	if [ -d arch/${ARCH}/include/generated ]; then
+		mkdir -p $kerneldir/arch/${ARCH}/include/generated/
+		cp -fR arch/${ARCH}/include/generated/* $kerneldir/arch/${ARCH}/include/generated/
+	fi
 }
 do_install[prefuncs] += "package_get_auto_pr"
 
-- 
2.1.0

[PATCH 3/7] kerneldev: create kernel-devsrc packaging

[Bruce Ashfield]

kernel-devsrc is responsible for creating and a packaging an environment
appropriate for kernel development (on or off target).

To create this support, we only need to copy/install the results of the
virtual/kernel providers build in the staging dir ... with some minor
manipulations to the source tree (.git removal and a clean up). This
produces a source tree that is capable of rebuilding the kernel on the
target.

Installing the kernel-devsrc package on a target (along with a
toolchain) is all that remains to be done.

 $ cd /usr/src/kernel
 $ make oldconfig
 $ make -j2 bzImage

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/recipes-kernel/linux/kernel-devsrc.bb | 50 ++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 meta/recipes-kernel/linux/kernel-devsrc.bb

diff --git a/meta/recipes-kernel/linux/kernel-devsrc.bb b/meta/recipes-kernel/linux/kernel-devsrc.bb
new file mode 100644
index 000000000000..facc235e2d0e
--- /dev/null
+++ b/meta/recipes-kernel/linux/kernel-devsrc.bb
@@ -0,0 +1,50 @@
+SUMMARY = "Linux kernel Development Source"
+DESCRIPTION = "Development source linux kernel. When built, this recipe packages the \
+source of the preferred virtual/kernel provider and makes it available for full kernel \
+development or external module builds"
+
+SECTION = "kernel"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/GPL-2.0;md5=801f80980d171dd6425610833a22dbe6"
+
+inherit linux-kernel-base
+
+# We need the kernel to be staged (unpacked, patched and configured) before
+# we can grab the source and make the kernel-devsrc package
+do_install[depends] += "virtual/kernel:do_populate_sysroot"
+
+# There's nothing to do here, except install the source where we can package it
+do_fetch[noexec] = "1"
+do_unpack[noexec] = "1"
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_populate_sysroot[noexec] = "1"
+
+# Define where the kernel headers are installed on the target as well as where
+# they are staged.
+KERNEL_SRC_PATH = "/usr/src/kernel"
+S = "${STAGING_DIR_TARGET}/${KERNEL_SRC_PATH}"
+
+KERNEL_VERSION = "${@get_kernelversion_headers('${S}')}"
+
+PACKAGE_ARCH = "${MACHINE_ARCH}"
+
+do_install() {
+        kerneldir=${D}${KERNEL_SRC_PATH}
+        install -d $kerneldir
+
+        #
+        # Copy the staging dir source (and module build support) into the devsrc structure.
+        # We can keep this copy simple and take everything, since a we'll clean up any build
+        # artifacts afterwards, and the extra i/o is not significant
+        #
+        cd ${S}
+        find . -type d -name '.git*' -prune -o -type f -print0 | cpio --null -pdlu $kerneldir
+        oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts
+}
+
+PACKAGES = "kernel-devsrc"
+FILES_${PN} = "${KERNEL_SRC_PATH}"
+RDEPENDS_${PN} = "bc"
-- 
2.1.0

[PATCH 4/7] images: introduce core-image-kernel-dev

[Bruce Ashfield]

When building a new kernel, or testing and update to the kernel there
are a set of packages that are sensitive to APIs, build system and
other changes associated with the kernel.

After building this recipe, we can be reasonably sure that a new kernel
package and coupled userspace have been built, installed and tested
via a single image.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/recipes-extended/images/core-image-kernel-dev.bb | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
 create mode 100644 meta/recipes-extended/images/core-image-kernel-dev.bb

diff --git a/meta/recipes-extended/images/core-image-kernel-dev.bb b/meta/recipes-extended/images/core-image-kernel-dev.bb
new file mode 100644
index 000000000000..d14f658726ab
--- /dev/null
+++ b/meta/recipes-extended/images/core-image-kernel-dev.bb
@@ -0,0 +1,17 @@
+DESCRIPTION = "A development image that builds the kernel and packages that are \
+sensitive to kernel updates and version changes"
+
+# Could also be core-image-basic, but we'll keep this small for now
+require recipes-core/images/core-image-minimal.bb
+
+KERNEL_DEV_UTILS ?= "dropbear"
+KERNEL_DEV_TOOLS ?= "packagegroup-core-tools-profile packagegroup-core-buildessential kernel-devsrc"
+KERNEL_DEV_MODULE ?= ""
+
+CORE_IMAGE_EXTRA_INSTALL += "${KERNEL_DEV_MODULE} \
+                             ${KERNEL_DEV_UTILS} \
+                             ${KERNEL_DEV_TOOLS} \
+                            "
+
+# We need extra space for things like kernel builds, etc.
+IMAGE_ROOTFS_EXTRA_SPACE_append += "+ 3000000"
-- 
2.1.0

[PATCH 5/7] lttng/perf: depend on virtual/kernel:do_install

[Bruce Ashfield]

To ensure that the minimal environment has been created for out of
tree module builds, perf and lttng should depend on the do_install
task of the kernel.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/recipes-kernel/lttng/lttng-modules_2.5.0.bb | 1 +
 meta/recipes-kernel/perf/perf.bb                 | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-kernel/lttng/lttng-modules_2.5.0.bb b/meta/recipes-kernel/lttng/lttng-modules_2.5.0.bb
index 6ff961a9b189..f66118011afe 100644
--- a/meta/recipes-kernel/lttng/lttng-modules_2.5.0.bb
+++ b/meta/recipes-kernel/lttng/lttng-modules_2.5.0.bb
@@ -7,6 +7,7 @@ LIC_FILES_CHKSUM = "file://LICENSE;md5=1412caf5a1aa90d6a48588a4794c0eac \
                     file://lgpl-2.1.txt;md5=243b725d71bb5df4a1e5920b344b86ad"
 
 DEPENDS = "virtual/kernel"
+do_configure[depends] += "virtual/kernel:do_install"
 
 inherit module
 
diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
index d375e246ce30..a42aa73c53c9 100644
--- a/meta/recipes-kernel/perf/perf.bb
+++ b/meta/recipes-kernel/perf/perf.bb
@@ -32,7 +32,7 @@ DEPENDS = " \
     bison flex \
 "
 
-do_configure[depends] += "virtual/kernel:do_patch"
+do_configure[depends] += "virtual/kernel:do_install"
 
 PROVIDES = "virtual/perf"
 
-- 
2.1.0

[PATCH 6/7] kernel-yocto: fix non-git builds

[Bruce Ashfield]

The ability to build non-git repositories was broken by two changes:

 - The existence of an empty 'patches' directory created during the
   unpack phase. This dir was incorrectly identified as a valid meta
   directory and broke the build. By ensuring that it is removed before
   creating the empty repository, it will no longer be found instead of
   the real meta directory.

 - The attempt to reset the git repository to a specific SRCREV when
   no SRCREV was provided. By checking for a SRCREV of 'INVALID', we
   avoid any processing and failed git operations.

Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/classes/kernel-yocto.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 45bdc9670afe..5706a9e4e338 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -179,9 +179,11 @@ do_kernel_checkout() {
 			bberror "S is not set to the linux source directory. Check "
 			bbfatal "the recipe and set S to the proper extracted subdirectory"
 		fi
+		rm -f .gitignore
 		git init
 		git add .
 		git commit -q -m "baseline commit: creating repo for ${PN}-${PV}"
+		git clean -d -f
 	fi
 	# end debare
 
@@ -287,7 +289,7 @@ do_validate_branches() {
 	if [ "${machine_srcrev}" = "AUTOINC" ]; then
 		bbnote "SRCREV validation is not required for AUTOREV"
 	elif [ "${machine_srcrev}" = "" ]; then
-		if [ "${SRCREV}" != "AUTOINC" ]; then
+		if [ "${SRCREV}" != "AUTOINC" ] && [ "${SRCREV}" != "INVALID" ]; then
 		       # SRCREV_machine_<MACHINE> was not set. This means that a custom recipe
 		       # that doesn't use the SRCREV_FORMAT "machine_meta" is being built. In
 		       # this case, we need to reset to the give SRCREV before heading to patching
-- 
2.1.0

[PATCH 7/7] kernel-yocto: make sure git tags get dereferenced properly in do_patch()

[Bruce Ashfield]

From: Jeff Wang <jeffrey.wang@mit.ll.edu>

Commit 92c1ece6c347030d48995a36f4c67861356e59d3 causes the test in do_patch()
in kernel-yocto.bbclass to fail if ${machine_srcrev} is an annotated tag. The
check is meant to ensure that ${machine_srcrev} is an ancestor of HEAD, but
if ${machine_srcrev} is a tag, then "$(git rev-parse --verify
${machine_srcrev})" evaluates to the SHA of the tag instead of what it's
pointing to.

Replacing "$(git rev-parse --verify ${machine_srcrev})" with "$(git rev-parse
--verify ${machine_srcrev}~0)" fixed the problem by finding the object pointed
to by the tag, and not the tag itself. This also works for commit IDs that
are not tags, hence is safe in a scenarios.

Jeff Wang <jeffrey.wang@ll.mit.edu>
Signed-off-by: Bruce Ashfield <bruce.ashfield@windriver.com>
---
 meta/classes/kernel-yocto.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index 5706a9e4e338..3b59d85f4ece 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -129,7 +129,7 @@ do_patch() {
 	# check to see if the specified SRCREV is reachable from the final branch.
 	# if it wasn't something wrong has happened, and we should error.
 	if [ "${machine_srcrev}" != "AUTOINC" ]; then
-		if ! [ "$(git rev-parse --verify ${machine_srcrev})" = "$(git merge-base ${machine_srcrev} HEAD)" ]; then
+		if ! [ "$(git rev-parse --verify ${machine_srcrev}~0)" = "$(git merge-base ${machine_srcrev} HEAD)" ]; then
 			bberror "SRCREV ${machine_srcrev} was specified, but is not reachable"
 			bbfatal "Check the BSP description for incorrect branch selection, or other errors."
 		fi
-- 
2.1.0

Re: [PATCH 1/7] kernel: Rearrange for 1.8

[Bruce Ashfield]

On Wed, Dec 10, 2014 at 5:34 AM, Bruce Ashfield
<bruce.ashfield@windriver.com> wrote:
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> In 1.8 we want to streamline the kernel build process. Basically we
> currently have multiple copies of the kernel source floating around
> and the copying/compression/decompression is painful.
>
> Lets assume we have a kernel source per machine since in most cases
> this is true (and we have a sysroot per machine anyway). Basically,
> instead of extracting a source into WORKDIR, then copying to a sysroot,
> we now set S to point straight at STAGING_DIR_KERNEL.
>
> Anything using kernel source can then just point at it and use:
>
> do_configure[depends] += "virtual/kernel:do_patch"
>
> to depend on the kernel source being present. Note this is different
> behaviour to DEPENDS += "virtual/kernel" which equates to
> do_configure[depends] += "virtual/kernel:do_populate_sysroot".
>
> Once we do this, we no longer need the copy operation in
> do_populate_sysroot, in fact there is nothing to do there (yay).
>
> The remaining part of the challenge is to kill off the horrible
> do_install. This patch splits it off to a different class, the idea here
> is to have a separate recipe which depends on the virtual/kernel:do_patch
> and just installs and packages the source needed to build modules on
> target into a specific package.
>
> Right now this code is proof of concept. It builds kernels and kernel
> modules. perf blows up in do_package with issues on finding the kernel
> version which can probably be fixed by adding back the right bit of do_install,
> and adding a dependency of do_package[depends] += "virtual/kernel:do_install"
> to perf. The whole thing needs a good write up, the corner cases testing
> and probably a good dose of cleanup to the remaining code.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/classes/kernel-yocto.bbclass      |  2 -
>  meta/classes/kernel.bbclass            | 91 ++--------------------------------
>  meta/classes/linux-kernel-base.bbclass | 13 ++++-
>  meta/recipes-kernel/perf/perf.bb       | 12 ++---
>  4 files changed, 20 insertions(+), 98 deletions(-)
>
> diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
> index c3e24943bbaf..45bdc9670afe 100644
> --- a/meta/classes/kernel-yocto.bbclass
> +++ b/meta/classes/kernel-yocto.bbclass
> @@ -1,5 +1,3 @@
> -S = "${WORKDIR}/linux"
> -
>  # remove tasks that modify the source tree in case externalsrc is inherited
>  SRCTREECOVEREDTASKS += "do_kernel_link_vmlinux do_kernel_configme do_validate_branches do_kernel_configcheck do_kernel_checkout do_patch"
>
> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> index fa06c3aae6f3..0d335a589121 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -3,6 +3,8 @@ inherit linux-kernel-base kernel-module-split
>  PROVIDES += "virtual/kernel"
>  DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native depmodwrapper-cross bc-native"
>
> +S = "${STAGING_DIR_TARGET}/${KERNEL_SRC_PATH}"
> +
>  # we include gcc above, we dont need virtual/libc
>  INHIBIT_DEFAULT_DEPS = "1"
>
> @@ -55,7 +57,7 @@ KERNEL_IMAGEDEST = "boot"
>  #
>  export CMDLINE_CONSOLE = "console=${@d.getVar("KERNEL_CONSOLE",1) or "ttyS0"}"
>
> -KERNEL_VERSION = "${@get_kernelversion('${B}')}"
> +KERNEL_VERSION = "${@get_kernelversion_headers('${B}')}"
>
>  KERNEL_LOCALVERSION ?= ""
>
> @@ -221,93 +223,6 @@ kernel_do_install() {
>         #
>
>         echo "${KERNEL_VERSION}" > $kerneldir/kernel-abiversion
> -
> -       #
> -       # Store kernel image name to allow use during image generation
> -       #
> -
> -       echo "${KERNEL_IMAGE_BASE_NAME}" >$kerneldir/kernel-image-name
> -
> -       #
> -       # Copy the entire source tree. In case an external build directory is
> -       # used, copy the build directory over first, then copy over the source
> -       # dir. This ensures the original Makefiles are used and not the
> -       # redirecting Makefiles in the build directory.
> -       #
> -       find . -depth -not -name "*.cmd" -not -name "*.o" -not -name "*.so.dbg" -not -name "*.so" -not -path "./Documentation*" -not -path "./source*" -not -path "./.*" -print0 | cpio --null -pdlu $kerneldir
> -       cp .config $kerneldir
> -       if [ "${S}" != "${B}" ]; then
> -               pwd="$PWD"
> -               cd "${S}"
> -               find . -depth -not -path "./Documentation*" -not -path "./.*" -print0 | cpio --null -pdlu $kerneldir
> -               cd "$pwd"
> -       fi
> -
> -       # Test to ensure that the output file and image type are not actually
> -       # the same file. If hardlinking is used, they will be the same, and there's
> -       # no need to install.
> -       ! [ ${KERNEL_OUTPUT} -ef $kerneldir/${KERNEL_IMAGETYPE} ] && install -m 0644 ${KERNEL_OUTPUT} $kerneldir/${KERNEL_IMAGETYPE}
> -       install -m 0644 System.map $kerneldir/System.map-${KERNEL_VERSION}
> -
> -       # Dummy Makefile so the clean below works
> -        mkdir $kerneldir/Documentation
> -        touch $kerneldir/Documentation/Makefile
> -
> -       #
> -       # Clean and remove files not needed for building modules.
> -       # Some distributions go through a lot more trouble to strip out
> -       # unecessary headers, for now, we just prune the obvious bits.
> -       #
> -       # We don't want to leave host-arch binaries in /sysroots, so
> -       # we clean the scripts dir while leaving the generated config
> -       # and include files.
> -       #
> -       oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean _mrproper_scripts
> -
> -       # hide directories that shouldn't have their .c, s and S files deleted
> -       for d in tools scripts lib; do
> -               mv $kerneldir/$d $kerneldir/.$d
> -       done
> -
> -       # delete .c, .s and .S files, unless we hid a directory as .<dir>. This technique is
> -       # much faster than find -prune and -exec
> -       find $kerneldir -not -path '*/\.*' -type f -name "*.[csS]" -delete
> -
> -       # put the hidden dirs back
> -       for d in tools scripts lib; do
> -               mv $kerneldir/.$d $kerneldir/$d
> -       done
> -
> -       # As of Linux kernel version 3.0.1, the clean target removes
> -       # arch/powerpc/lib/crtsavres.o which is present in
> -       # KBUILD_LDFLAGS_MODULE, making it required to build external modules.
> -       if [ ${ARCH} = "powerpc" ]; then
> -               cp -l arch/powerpc/lib/crtsavres.o $kerneldir/arch/powerpc/lib/crtsavres.o
> -       fi
> -
> -       # Necessary for building modules like compat-wireless.
> -       if [ -f include/generated/bounds.h ]; then
> -               cp -l include/generated/bounds.h $kerneldir/include/generated/bounds.h
> -       fi
> -       if [ -d arch/${ARCH}/include/generated ]; then
> -               mkdir -p $kerneldir/arch/${ARCH}/include/generated/
> -               cp -flR arch/${ARCH}/include/generated/* $kerneldir/arch/${ARCH}/include/generated/
> -       fi
> -
> -       # Remove the following binaries which cause strip or arch QA errors
> -       # during do_package for cross-compiled platforms
> -       bin_files="arch/powerpc/boot/addnote arch/powerpc/boot/hack-coff \
> -                  arch/powerpc/boot/mktree scripts/kconfig/zconf.tab.o \
> -                  scripts/kconfig/conf.o scripts/kconfig/kxgettext.o"
> -       for entry in $bin_files; do
> -               rm -f $kerneldir/$entry
> -       done
> -
> -       # kernels <2.6.30 don't have $kerneldir/tools directory so we check if it exists before calling sed
> -       if [ -f $kerneldir/tools/perf/Makefile ]; then
> -               # Fix SLANG_INC for slang.h
> -               sed -i 's#-I/usr/include/slang#-I=/usr/include/slang#g' $kerneldir/tools/perf/Makefile
> -       fi
>  }
>  do_install[prefuncs] += "package_get_auto_pr"
>
> diff --git a/meta/classes/linux-kernel-base.bbclass b/meta/classes/linux-kernel-base.bbclass
> index 4f2b0a4a9844..89ce71605c00 100644
> --- a/meta/classes/linux-kernel-base.bbclass
> +++ b/meta/classes/linux-kernel-base.bbclass
> @@ -1,5 +1,5 @@
>  # parse kernel ABI version out of <linux/version.h>
> -def get_kernelversion(p):
> +def get_kernelversion_headers(p):
>      import re
>
>      fn = p + '/include/linux/utsrelease.h'
> @@ -9,7 +9,6 @@ def get_kernelversion(p):
>      if not os.path.isfile(fn):
>          fn = p + '/include/linux/version.h'
>
> -    import re
>      try:
>          f = open(fn, 'r')
>      except IOError:
> @@ -24,6 +23,16 @@ def get_kernelversion(p):
>              return m.group(1)
>      return None
>
> +
> +def get_kernelversion_file(p):
> +    fn = p + '/kernel-abiversion'
> +
> +    try:
> +        with open(fn, 'r') as f:
> +            return f.readlines()[0].strip()
> +    except IOError:
> +        return None
> +
>  def linux_module_packages(s, d):
>         suffix = ""
>         return " ".join(map(lambda s: "kernel-module-%s%s" % (s.lower().replace('_', '-').replace('@', '+'), suffix), s.split()))
> diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
> index 3771cdb3e9dd..d375e246ce30 100644
> --- a/meta/recipes-kernel/perf/perf.bb
> +++ b/meta/recipes-kernel/perf/perf.bb
> @@ -22,7 +22,7 @@ TUI_DEPENDS = "${@perf_feature_enabled('perf-tui', 'libnewt slang', '',d)}"
>  SCRIPTING_DEPENDS = "${@perf_feature_enabled('perf-scripting', 'perl python', '',d)}"
>  LIBUNWIND_DEPENDS = "${@perf_feature_enabled('perf-libunwind', 'libunwind', '',d)}"
>
> -DEPENDS = "virtual/kernel \
> +DEPENDS = " \
>      virtual/${MLPREFIX}libc \
>      ${MLPREFIX}elfutils \
>      ${MLPREFIX}binutils \
> @@ -32,6 +32,8 @@ DEPENDS = "virtual/kernel \
>      bison flex \
>  "
>
> +do_configure[depends] += "virtual/kernel:do_patch"
> +
>  PROVIDES = "virtual/perf"
>
>  inherit linux-kernel-base kernel-arch pythonnative
> @@ -46,7 +48,7 @@ export HOST_SYS
>  #kernel 3.1+ supports WERROR to disable warnings as errors
>  export WERROR = "0"
>
> -do_populate_lic[depends] += "virtual/kernel:do_populate_sysroot"
> +do_populate_lic[depends] += "virtual/kernel:do_patch"
>
>  # needed for building the tools/perf Perl binding
>  inherit perlnative cpan-base
> @@ -56,9 +58,7 @@ export PERL_INC = "${STAGING_LIBDIR}${PERL_OWN_DIR}/perl/${@get_perl_version(d)}
>  export PERL_LIB = "${STAGING_LIBDIR}${PERL_OWN_DIR}/perl/${@get_perl_version(d)}"
>  export PERL_ARCHLIB = "${STAGING_LIBDIR}${PERL_OWN_DIR}/perl/${@get_perl_version(d)}"
>
> -S = "${STAGING_KERNEL_DIR}"
> -# The source should be ready after the do_unpack
> -do_unpack[depends] += "virtual/kernel:do_populate_sysroot"
> +inherit kernelsrc

When juggling the patches .. I managed to leave this in .. when it should
have been removed. I'm fixing up this patch to no longer inherit this
class, since
it is no longer needed.

Bruce

>
>  B = "${WORKDIR}/${BPN}-${PV}"
>
> @@ -157,7 +157,7 @@ do_configure_prepend () {
>  }
>
>  python do_package_prepend() {
> -    bb.data.setVar('PKGV', get_kernelversion('${S}').split("-")[0], d)
> +    d.setVar('PKGV', d.getVar("KERNEL_VERSION", True).split("-")[0])
>  }
>
>  PACKAGE_ARCH = "${MACHINE_ARCH}"
> --
> 2.1.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Enrico Scholz]

Bruce Ashfield <bruce.ashfield-CWA4WttNNZF54TAoqtyWWQ@public.gmane.org>
writes:

> With the kernel build optimizations, we no longer copy the source from
> the built kernel into the staging dir, since the kernel is unpacked and built
> directly from the staging dir.
>
> This means that a few build artifacts need to be restored to allow out of
> tree modules to once again build.
> ...
> +	
> +	# Copy files required for module builds
> +	cp System.map $kerneldir/System.map-${KERNEL_VERSION}
> +	cp Module.symvers $kerneldir/
> +	cp .config $kerneldir/

This and the following lines seem to void the whole idea of the patchset
because it means that only one kernel can be built.  In our usecase we
build usually two kernels: a full featured one for the main system and a
minimal, initramfs based for a rescue system.

With this change and building both kernels, it is ambiguous which
System.map is used.



Enrico
-- 
SIGMA Chemnitz GmbH       Registergericht:   Amtsgericht Chemnitz HRB 1750
Am Erlenwald 13           Geschaeftsfuehrer: Grit Freitag, Frank Pyritz
09128 Chemnitz

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Richard Purdie]

On Fri, 2014-12-19 at 12:05 +0100, Enrico Scholz wrote:
> Bruce Ashfield <bruce.ashfield-CWA4WttNNZF54TAoqtyWWQ@public.gmane.org>
> writes:
> 
> > With the kernel build optimizations, we no longer copy the source from
> > the built kernel into the staging dir, since the kernel is unpacked and built
> > directly from the staging dir.
> >
> > This means that a few build artifacts need to be restored to allow out of
> > tree modules to once again build.
> > ...
> > +	
> > +	# Copy files required for module builds
> > +	cp System.map $kerneldir/System.map-${KERNEL_VERSION}
> > +	cp Module.symvers $kerneldir/
> > +	cp .config $kerneldir/
> 
> This and the following lines seem to void the whole idea of the patchset
> because it means that only one kernel can be built.  In our usecase we
> build usually two kernels: a full featured one for the main system and a
> minimal, initramfs based for a rescue system.
> 
> With this change and building both kernels, it is ambiguous which
> System.map is used.

Regardless of how you do this, if you build two kernels, you still need
to choose one to build the main set of modules against. I'm therefore
not sure how this change makes anything particularly worst than what
existed before?

Building two kernels is currently fraught with difficulty anyway, to the
point of not being a currently well supported use case and the classes
would probably need to be refactored to allow such things to be well
supported. These changes do bring significant benefit to the most common
case users so we probably need to find a way to allow your use case to
work with the new model.

Cheers,

Richard

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

>> In our usecase we build usually two kernels: a full featured one for
>> the main system and a minimal, initramfs based for a rescue system.
>> 
>> With this change and building both kernels, it is ambiguous which
>> System.map is used.
>
> Regardless of how you do this, if you build two kernels, you still need
> to choose one to build the main set of modules against. I'm therefore
> not sure how this change makes anything particularly worst than what
> existed before?

A I see... previously I did

| do_install[noexec] = "1"

in the rescue kernel and only the main kernel's System.map, .config... were
staged into the sysroot.  This can be done now too, but building secondary
kernels fails with

| .../usr/src/kernel is not clean, please run 'make mrproper'


> Building two kernels is currently fraught with difficulty anyway,

It was not so difficulty... Removing the "virtual/kernel" provider and
changing names of deployed files should do it with the old classes.


> to the point of not being a currently well supported use case and the
> classes would probably need to be refactored to allow such things to
> be well supported. These changes do bring significant benefit

Does some lowered i/o really provide such a benefit?  With the new
system you are bypassing the excellent staging system (which would
detect e g ".config" or "System.map" file conflicts). Afais, it breaks
with "rm_work" too (I guess, kernel:fetch stamp is removed when kernel
build finishes and packages which inherit "kernelsrc" fetch the kernel
sources (without .config and System.map) again).



Enrico

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Richard Purdie]

On Fri, 2014-12-19 at 13:11 +0100, Enrico Scholz wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> > to the point of not being a currently well supported use case and the
> > classes would probably need to be refactored to allow such things to
> > be well supported. These changes do bring significant benefit
> 
> Does some lowered i/o really provide such a benefit?  With the new
> system you are bypassing the excellent staging system (which would
> detect e g ".config" or "System.map" file conflicts). Afais, it breaks
> with "rm_work" too (I guess, kernel:fetch stamp is removed when kernel
> build finishes and packages which inherit "kernelsrc" fetch the kernel
> sources (without .config and System.map) again).

In summary, basically, yes. The kernel source is huge and we were
compressing/decompressing it in several places on the critical path.
People were struggling to develop kernels using the system due to the
overhead.

Whilst this approach does bypass some parts of the system, I do believe
the benefits are worth it. We're talking about making the kernel build
time about three times faster iirc, and reducing its sstate footprint
significantly.

Cheers,

Richard

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> In summary, basically, yes. The kernel source is huge and we were
> compressing/decompressing it in several places on the critical path.
> People were struggling to develop kernels using the system due to the
> overhead.

I do not see how the new system makes it easier to "develop kernels".  Due
to polluting sources with .config and related files, KBUILD_OUTPUT/VPATH
builds are not possible from this tree.

In my experience, it is also a pain to jump between the different
directories and tools like 'cscope' are not prepared of it.

And more importantly: the new system lowers end user experience
significantly because kernel messages will show absolute buildsystem
paths now; e.g.

| /srv/.oe/bld/e6ca2c38-c20d-f57f-7eca-ffc0aaa2f0bd/sysroots/kk-trizeps6/usr/src/kernel/drivers/usb/core/hub.c

vs.

| drivers/usb/core/hub.c

VPATH builds might be interesting for QA (e.g. building from same source
with different configuration) but should not be used for final builds.


> Whilst this approach does bypass some parts of the system, I do believe
> the benefits are worth it. We're talking about making the kernel build
> time about three times faster iirc,

I can not reproduce these numbers here; I get (after a '-c cleanall' and
'ccache -c'):

  | Task                         | time (old) | time (new) |
  |------------------------------+------------+------------|
  | do_bundle_initramfs          |   0.087052 |   0.034955 |
  | do_compile                   | 128.242407 | 133.723027 |
  | do_compile_kernelmodules     |  84.415655 |  83.249409 |
  | do_compile_prepare           |   2.885401 |   1.714159 |
  | do_configure                 |   6.202691 |   4.340526 |
  | do_deploy                    |  13.991785 |   14.07096 |
  | do_fetch                     |   0.210244 |   1.425304 |
  | do_generate_initramfs_source |   0.063915 |   0.041925 |
  | do_install                   |  16.190504 |    2.91906 |
  | do_package                   | 120.823374 |  16.422429 |
  | do_package_qa                |            |   2.557622 |
  | do_package_write_ipk         |   42.50694 |   29.57585 |
  | do_packagedata               |   1.612542 |   0.462001 |
  | do_patch                     |   0.186583 |   0.011232 |
  | do_populate_lic              |   0.795013 |   0.135186 |
  | do_populate_sysroot          |  10.142978 |   0.533519 |
  | do_rm_work                   |   1.762486 |   0.447187 |
  | do_rm_work_all               |   0.049144 |   0.030964 |
  | do_sizecheck                 |   0.054441 |   0.035806 |
  | do_strip                     |   0.049917 |   0.030841 |
  | do_uboot_mkimage             |   9.032543 |   12.83624 |
  | do_unpack                    |   6.695678 |   9.322173 |

  | old | 446.00129 |
  | new | 313.92038 |

Although the 'new' system is faster, this is gained mainly by the
'do_package' task which does not seem to be comparable.  The new
method will create only a very small 'kernel-dev' subpackage:

   1,1M    tmp/deploy/ipk/kk_trizeps6/kernel-dev_3.14...

vs.

    36M    tmp/deploy/ipk/kk_trizeps6/kernel-dev_3.14...

so the old task can be improved either by removing some files, or the
new task misses files.



Enrico

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Richard Purdie]

On Tue, 2014-12-23 at 03:07 +0100, Enrico Scholz wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> 
> > In summary, basically, yes. The kernel source is huge and we were
> > compressing/decompressing it in several places on the critical path.
> > People were struggling to develop kernels using the system due to the
> > overhead.
> 
> I do not see how the new system makes it easier to "develop kernels".

One of the complaints I keep hearing is how long the populate_sysroot
and package tasks take. I also get complaints about the sstate footprint
of those tasks.

Imagine you're doing a "bitbake virtual/kernel -c compile -f", hacking
on a module, you then package and install it onto the target to test.

As things stood, the package and populate_sysroot tasks took a while
(I'll get to times in a minute). This was particularly true with large
kernel source directories.

We also had issues about how and when "make scripts" should be run on
the kernel directory to enable module building and conflicts over people
wanting the source code to be able to develop modules on target. Not
everyone wants the latter but it should be available to those who do
want it and the old kernel-dev package was basically crippled.

Taking a step back from things and keeping in mind some of the
"developer experience" goals of the 1.8 release, we therefore looked at
how we could improve things. The biggest problem and overhead is the
amount of copying of a large chunk of data (the kernel source) that we
make and the fact that it was incomplete in kernel-dev due to wanting
speed (at the expense of breaking it). The changes therefore look at
improving things in that area.

>   Due
> to polluting sources with .config and related files, KBUILD_OUTPUT/VPATH
> builds are not possible from this tree.

Agreed and I'm actually not happy about this. I think going forward
we'll need a second directory for these so we do keep the output
separate and can recompile against it. Bruce, please take note we
probably need to change this.

> In my experience, it is also a pain to jump between the different
> directories and tools like 'cscope' are not prepared of it.
> 
> And more importantly: the new system lowers end user experience
> significantly because kernel messages will show absolute buildsystem
> paths now; e.g.
> 
> | /srv/.oe/bld/e6ca2c38-c20d-f57f-7eca-ffc0aaa2f0bd/sysroots/kk-trizeps6/usr/src/kernel/drivers/usb/core/hub.c
> 
> vs.
> 
> | drivers/usb/core/hub.c

See my other email on the two sides to this. Here I can tell from a cut
and pasted failure which revisions you were trying to build so it
actually makes the logs more useful in some contexts. That isn't the
reason we've done this but I'd say it can be helpful.

> VPATH builds might be interesting for QA (e.g. building from same source
> with different configuration) but should not be used for final builds.
> 
> 
> > Whilst this approach does bypass some parts of the system, I do believe
> > the benefits are worth it. We're talking about making the kernel build
> > time about three times faster iirc,

I did say "iirc" and I don't exactly remember the circumstances I tested
that, see below, sorry :(.

> I can not reproduce these numbers here; I get (after a '-c cleanall' and
> 'ccache -c'):
> 
>   | Task                         | time (old) | time (new) |
>   |------------------------------+------------+------------|
>   | do_bundle_initramfs          |   0.087052 |   0.034955 |
>   | do_compile                   | 128.242407 | 133.723027 |
>   | do_compile_kernelmodules     |  84.415655 |  83.249409 |
>   | do_compile_prepare           |   2.885401 |   1.714159 |
>   | do_configure                 |   6.202691 |   4.340526 |
>   | do_deploy                    |  13.991785 |   14.07096 |
>   | do_fetch                     |   0.210244 |   1.425304 |
>   | do_generate_initramfs_source |   0.063915 |   0.041925 |
>   | do_install                   |  16.190504 |    2.91906 |
>   | do_package                   | 120.823374 |  16.422429 |
>   | do_package_qa                |            |   2.557622 |
>   | do_package_write_ipk         |   42.50694 |   29.57585 |
>   | do_packagedata               |   1.612542 |   0.462001 |
>   | do_patch                     |   0.186583 |   0.011232 |
>   | do_populate_lic              |   0.795013 |   0.135186 |
>   | do_populate_sysroot          |  10.142978 |   0.533519 |
>   | do_rm_work                   |   1.762486 |   0.447187 |
>   | do_rm_work_all               |   0.049144 |   0.030964 |
>   | do_sizecheck                 |   0.054441 |   0.035806 |
>   | do_strip                     |   0.049917 |   0.030841 |
>   | do_uboot_mkimage             |   9.032543 |   12.83624 |
>   | do_unpack                    |   6.695678 |   9.322173 |
> 
>   | old | 446.00129 |
>   | new | 313.92038 |

So you have a gain here from 7.4 mins to 5.2 mins which isn't bad. I'd
observe your numbers suggest you have pretty fast disk IO, I've seen the
populate_sysroot task take a lot longer due to the amount of data being
copied around. Keep in mind that whilst do_package improved, so did
package_write_ipk, rm_work, install and populate_sysroot, all quite a
bit more than "three times".

I suspect my "three times" number comes from my highly parallel core
system where the do_compile/do_compilemodules step runs extremely
quickly (say 20s) but do_package was about the same as this so you can
imagine if you reduce do_package from 120s to 16s, you could see an
increase of about "three times".

To give another perspective on the times, we run "official" benchmarks
of some things and the line before and after the changes merged from
that log is:

fedora19,master:88528a128fe7c11871c24706ff6d245b183d6975,1.7_M2-1510-g88528a1,1:18:03,11:16.26,1:13:35,4:11.18,0:34.68,0:15.72,0:01.08,24805964,5548960
fedora19,master:b99419ff4c1b5ac1e4acd34d95fffd3ac5458bad,1.7_M2-1553-gb99419f,1:13:20,8:10.41,1:10:07,4:08.31,0:34.67,0:15.77,0:01.05,24385880,5956872

What that shows is that:

The "bitbake core-image-sato -c rootfs" time without rm_work went from
1:18 to 1:13, i.e. improved 5 mins.

The "bitbake virtual/kernel" time went from 11:16 to 8:10, i.e. improved
by 3 mins.

The "bitbake core-image-sato -c rootfs" time with rm_work went from 1:13
to 1:10, i.e. improved 3 mins.

and the size of the build with rm_work increased by around 500MB, the
size of the build without rm_work decreased by 500MB.

I will also add that we're not done yet with some of these tweaks. I
know linux-yocto is doing some things in do_patch that are performance
intensive and we plan to fix those which will further improve the 11
mins number above.

> Although the 'new' system is faster, this is gained mainly by the
> 'do_package' task which does not seem to be comparable.  The new
> method will create only a very small 'kernel-dev' subpackage:
> 
>    1,1M    tmp/deploy/ipk/kk_trizeps6/kernel-dev_3.14...
> 
> vs.
> 
>     36M    tmp/deploy/ipk/kk_trizeps6/kernel-dev_3.14...
> 
> so the old task can be improved either by removing some files, or the
> new task misses files.

There is a new kernel-devsrc package which contains some things that
used to be in kernel-dev. It is way more complete and functional than
the old one ever was since the old one was a hacked up copy of the
kernel source. It also only gets built if you request it, only a small
number of people need it and its hence much more user friendly this way.

I am sorry you're not happy about the changes :(.

Cheers,

Richard

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

>> > In summary, basically, yes. The kernel source is huge and we were
>> > compressing/decompressing it in several places on the critical
>> > path.  People were struggling to develop kernels using the system
>> > due to the overhead.
>> 
>> I do not see how the new system makes it easier to "develop kernels".
>
> One of the complaints I keep hearing is how long the populate_sysroot
> and package tasks take. I also get complaints about the sstate footprint
> of those tasks.

Perhaps, it should be possible to disable sstate'ing for certain recipes?
In case of the new kernel build system, you swap the expensive sstate'ing
with an expensive unpacking.


> Imagine you're doing a "bitbake virtual/kernel -c compile -f", hacking
> on a module, you then package and install it onto the target to test.

1. Are there really people who are doing such things?  There are much
   more effective ways to do kernel development with OE (you have the
   overhead from bitbake reading its database; do_compile() does not
   deploy images on TFTP server nor the modules in the NFS sysroot).

2. This usecase is not supported by the current classes:

   1. the 'bitbake virtual/kernel -c compile -f' removes .config and
      System.map from the kernel sources (else, do_compile() will fail)

   2. trying to build the module (with 'M=...')will fail because .config
      + System.map are missing in the source


>> to polluting sources with .config and related files, KBUILD_OUTPUT/VPATH
>> builds are not possible from this tree.
>
> Agreed and I'm actually not happy about this. I think going forward
> we'll need a second directory for these so we do keep the output
> separate and can recompile against it.

'cp -al' a template kernel and .config etc. into ${WORKDIR} should be
pretty fast.


Enrico

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Bruce Ashfield]

On Tue, Dec 23, 2014 at 4:54 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2014-12-23 at 03:07 +0100, Enrico Scholz wrote:
>> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>>
>> > In summary, basically, yes. The kernel source is huge and we were
>> > compressing/decompressing it in several places on the critical path.
>> > People were struggling to develop kernels using the system due to the
>> > overhead.
>>
>> I do not see how the new system makes it easier to "develop kernels".
>
> One of the complaints I keep hearing is how long the populate_sysroot
> and package tasks take. I also get complaints about the sstate footprint
> of those tasks.
>
> Imagine you're doing a "bitbake virtual/kernel -c compile -f", hacking
> on a module, you then package and install it onto the target to test.
>
> As things stood, the package and populate_sysroot tasks took a while
> (I'll get to times in a minute). This was particularly true with large
> kernel source directories.
>
> We also had issues about how and when "make scripts" should be run on
> the kernel directory to enable module building and conflicts over people
> wanting the source code to be able to develop modules on target. Not
> everyone wants the latter but it should be available to those who do
> want it and the old kernel-dev package was basically crippled.
>
> Taking a step back from things and keeping in mind some of the
> "developer experience" goals of the 1.8 release, we therefore looked at
> how we could improve things. The biggest problem and overhead is the
> amount of copying of a large chunk of data (the kernel source) that we
> make and the fact that it was incomplete in kernel-dev due to wanting
> speed (at the expense of breaking it). The changes therefore look at
> improving things in that area.
>
>>   Due
>> to polluting sources with .config and related files, KBUILD_OUTPUT/VPATH
>> builds are not possible from this tree.
>
> Agreed and I'm actually not happy about this. I think going forward
> we'll need a second directory for these so we do keep the output
> separate and can recompile against it. Bruce, please take note we
> probably need to change this.


Noted. And I have a patch in flight for this. I started it before leaving
for the holidays, but with the other build issues, I didn't get a chance to
complete it. This is priority #1 either over the holidays (unlikely, but I
can hope) or the first weeks of January.

Bruce

>
>> In my experience, it is also a pain to jump between the different
>> directories and tools like 'cscope' are not prepared of it.
>>
>> And more importantly: the new system lowers end user experience
>> significantly because kernel messages will show absolute buildsystem
>> paths now; e.g.
>>
>> | /srv/.oe/bld/e6ca2c38-c20d-f57f-7eca-ffc0aaa2f0bd/sysroots/kk-trizeps6/usr/src/kernel/drivers/usb/core/hub.c
>>
>> vs.
>>
>> | drivers/usb/core/hub.c
>
> See my other email on the two sides to this. Here I can tell from a cut
> and pasted failure which revisions you were trying to build so it
> actually makes the logs more useful in some contexts. That isn't the
> reason we've done this but I'd say it can be helpful.
>
>> VPATH builds might be interesting for QA (e.g. building from same source
>> with different configuration) but should not be used for final builds.
>>
>>
>> > Whilst this approach does bypass some parts of the system, I do believe
>> > the benefits are worth it. We're talking about making the kernel build
>> > time about three times faster iirc,
>
> I did say "iirc" and I don't exactly remember the circumstances I tested
> that, see below, sorry :(.
>
>> I can not reproduce these numbers here; I get (after a '-c cleanall' and
>> 'ccache -c'):
>>
>>   | Task                         | time (old) | time (new) |
>>   |------------------------------+------------+------------|
>>   | do_bundle_initramfs          |   0.087052 |   0.034955 |
>>   | do_compile                   | 128.242407 | 133.723027 |
>>   | do_compile_kernelmodules     |  84.415655 |  83.249409 |
>>   | do_compile_prepare           |   2.885401 |   1.714159 |
>>   | do_configure                 |   6.202691 |   4.340526 |
>>   | do_deploy                    |  13.991785 |   14.07096 |
>>   | do_fetch                     |   0.210244 |   1.425304 |
>>   | do_generate_initramfs_source |   0.063915 |   0.041925 |
>>   | do_install                   |  16.190504 |    2.91906 |
>>   | do_package                   | 120.823374 |  16.422429 |
>>   | do_package_qa                |            |   2.557622 |
>>   | do_package_write_ipk         |   42.50694 |   29.57585 |
>>   | do_packagedata               |   1.612542 |   0.462001 |
>>   | do_patch                     |   0.186583 |   0.011232 |
>>   | do_populate_lic              |   0.795013 |   0.135186 |
>>   | do_populate_sysroot          |  10.142978 |   0.533519 |
>>   | do_rm_work                   |   1.762486 |   0.447187 |
>>   | do_rm_work_all               |   0.049144 |   0.030964 |
>>   | do_sizecheck                 |   0.054441 |   0.035806 |
>>   | do_strip                     |   0.049917 |   0.030841 |
>>   | do_uboot_mkimage             |   9.032543 |   12.83624 |
>>   | do_unpack                    |   6.695678 |   9.322173 |
>>
>>   | old | 446.00129 |
>>   | new | 313.92038 |
>
> So you have a gain here from 7.4 mins to 5.2 mins which isn't bad. I'd
> observe your numbers suggest you have pretty fast disk IO, I've seen the
> populate_sysroot task take a lot longer due to the amount of data being
> copied around. Keep in mind that whilst do_package improved, so did
> package_write_ipk, rm_work, install and populate_sysroot, all quite a
> bit more than "three times".
>
> I suspect my "three times" number comes from my highly parallel core
> system where the do_compile/do_compilemodules step runs extremely
> quickly (say 20s) but do_package was about the same as this so you can
> imagine if you reduce do_package from 120s to 16s, you could see an
> increase of about "three times".
>
> To give another perspective on the times, we run "official" benchmarks
> of some things and the line before and after the changes merged from
> that log is:
>
> fedora19,master:88528a128fe7c11871c24706ff6d245b183d6975,1.7_M2-1510-g88528a1,1:18:03,11:16.26,1:13:35,4:11.18,0:34.68,0:15.72,0:01.08,24805964,5548960
> fedora19,master:b99419ff4c1b5ac1e4acd34d95fffd3ac5458bad,1.7_M2-1553-gb99419f,1:13:20,8:10.41,1:10:07,4:08.31,0:34.67,0:15.77,0:01.05,24385880,5956872
>
> What that shows is that:
>
> The "bitbake core-image-sato -c rootfs" time without rm_work went from
> 1:18 to 1:13, i.e. improved 5 mins.
>
> The "bitbake virtual/kernel" time went from 11:16 to 8:10, i.e. improved
> by 3 mins.
>
> The "bitbake core-image-sato -c rootfs" time with rm_work went from 1:13
> to 1:10, i.e. improved 3 mins.
>
> and the size of the build with rm_work increased by around 500MB, the
> size of the build without rm_work decreased by 500MB.
>
> I will also add that we're not done yet with some of these tweaks. I
> know linux-yocto is doing some things in do_patch that are performance
> intensive and we plan to fix those which will further improve the 11
> mins number above.
>
>> Although the 'new' system is faster, this is gained mainly by the
>> 'do_package' task which does not seem to be comparable.  The new
>> method will create only a very small 'kernel-dev' subpackage:
>>
>>    1,1M    tmp/deploy/ipk/kk_trizeps6/kernel-dev_3.14...
>>
>> vs.
>>
>>     36M    tmp/deploy/ipk/kk_trizeps6/kernel-dev_3.14...
>>
>> so the old task can be improved either by removing some files, or the
>> new task misses files.
>
> There is a new kernel-devsrc package which contains some things that
> used to be in kernel-dev. It is way more complete and functional than
> the old one ever was since the old one was a hacked up copy of the
> kernel source. It also only gets built if you request it, only a small
> number of people need it and its hence much more user friendly this way.
>
> I am sorry you're not happy about the changes :(.
>
> Cheers,
>
> Richard
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH 2/7] kernel: fix out of tree module builds

[Bruce Ashfield]

On Tue, Dec 23, 2014 at 6:51 AM, Enrico Scholz
<enrico.scholz@sigma-chemnitz.de> wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>
>>> > In summary, basically, yes. The kernel source is huge and we were
>>> > compressing/decompressing it in several places on the critical
>>> > path.  People were struggling to develop kernels using the system
>>> > due to the overhead.
>>>
>>> I do not see how the new system makes it easier to "develop kernels".
>>
>> One of the complaints I keep hearing is how long the populate_sysroot
>> and package tasks take. I also get complaints about the sstate footprint
>> of those tasks.
>
> Perhaps, it should be possible to disable sstate'ing for certain recipes?
> In case of the new kernel build system, you swap the expensive sstate'ing
> with an expensive unpacking.
>
>
>> Imagine you're doing a "bitbake virtual/kernel -c compile -f", hacking
>> on a module, you then package and install it onto the target to test.
>
> 1. Are there really people who are doing such things?  There are much
>    more effective ways to do kernel development with OE (you have the
>    overhead from bitbake reading its database; do_compile() does not
>    deploy images on TFTP server nor the modules in the NFS sysroot).

There are definitely people doing this. And if the build system wouldn't
get in the way .. more definitely could.

>
> 2. This usecase is not supported by the current classes:
>
>    1. the 'bitbake virtual/kernel -c compile -f' removes .config and
>       System.map from the kernel sources (else, do_compile() will fail)

This one is going to be fixed with the follow up patches I mentioned
in another email, since we can maintain the output artifacts separate
from ${B} and build against them, that keeps us working with rm_work
and keeps things light.

>
>    2. trying to build the module (with 'M=...')will fail because .config
>       + System.map are missing in the source

#2 shouldn't happen if the module depends on the kernel's do_install
(as we intended), since that puts the external module build artifacts
back in place.

>
>
>>> to polluting sources with .config and related files, KBUILD_OUTPUT/VPATH
>>> builds are not possible from this tree.
>>
>> Agreed and I'm actually not happy about this. I think going forward
>> we'll need a second directory for these so we do keep the output
>> separate and can recompile against it.
>
> 'cp -al' a template kernel and .config etc. into ${WORKDIR} should be
> pretty fast.

With a split source and build configuration .. this shouldn't be needed.
If we start copying these things back into WORKDIR .. then something
is wrong.

Bruce

>
>
> Enrico
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"