[PATCH 0/7][RFC][HEREBEDRAGONS] Remove bitbake's knowledge of ${B}

[PATCH 0/7][RFC][HEREBEDRAGONS] Remove bitbake's knowledge of ${B}

[Ross Burton]

Hi,

So when starting a new task if you don't tell bitbake what directories it should
create (and implicit, chdir() into) it uses ${B}.  Which is bitbake knowing
about openembedded variables, and generally frowned upon.

This branch series removes that knowledge, and if a task is executed without any
directories to create or chdir into it just doesn't do anything.

This revealed a few classes and recipes which assume they're being ran in ${B}
so the rest of the series is either adding back [dirs] flags to the tasks, or
making paths absolute.

Testing has been quite minimal: core-image-sato builds but I've not verified the
contents haven't changed yet.

Comments?

Ross


The following changes since commit 7f8286fb9ec9f9d9f606c69e5abd717078ace5b2:

  classes/sstate.bbclass: Improve checkstatus using connection cache. (2015-07-12 22:55:46 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ross/dirs

for you to fetch changes up to b306a4b740868835ff0e0ac14d8b7ed9afe83091:

  kernel: fix path assumptions (2015-07-13 14:51:57 +0100)

----------------------------------------------------------------
Ross Burton (7):
      build: don't automatically mkdir/chdir B
      build: add remove/mkdir/chdir debugging
      ptest: add dirs
      boost: add dirs
      python: add explicit directory when copying
      libc-package: fix path assumption
      kernel: fix path assumptions

 bitbake/lib/bb/build.py                       | 19 ++++++++++++++++++-
 meta/classes/kernel.bbclass                   |  8 ++++----
 meta/classes/libc-package.bbclass             |  3 +--
 meta/classes/ptest.bbclass                    |  3 +++
 meta/recipes-devtools/python/python3_3.4.3.bb |  2 +-
 meta/recipes-devtools/python/python_2.7.9.bb  |  2 +-
 meta/recipes-support/boost/boost.inc          |  1 +
 7 files changed, 29 insertions(+), 9 deletions(-)

Ross Burton (7):
  build: don't automatically mkdir/chdir B
  build: add remove/mkdir/chdir debugging
  ptest: add dirs
  boost: add dirs
  python: add explicit directory when copying
  libc-package: fix path assumption
  kernel: fix path assumptions

 bitbake/lib/bb/build.py                       | 19 ++++++++++++++++++-
 meta/classes/kernel.bbclass                   |  8 ++++----
 meta/classes/libc-package.bbclass             |  3 +--
 meta/classes/ptest.bbclass                    |  3 +++
 meta/recipes-devtools/python/python3_3.4.3.bb |  2 +-
 meta/recipes-devtools/python/python_2.7.9.bb  |  2 +-
 meta/recipes-support/boost/boost.inc          |  1 +
 7 files changed, 29 insertions(+), 9 deletions(-)

-- 
2.1.4

[PATCH 1/7] build: don't automatically mkdir/chdir B

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 bitbake/lib/bb/build.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index cce01fe..7cefee7 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -183,7 +183,9 @@ def exec_func(func, d, dirs = None):
         adir = dirs[-1]
     else:
         adir = d.getVar('B', True)
-        bb.utils.mkdirhier(adir)
+        bb.note("NOT creating implicit %s (for %s)" % (adir, func))
+        #bb.utils.mkdirhier(adir)
+        adir = None
 
     ispython = flags.get('python')
 
-- 
2.1.4

[PATCH 2/7] build: add remove/mkdir/chdir debugging

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 bitbake/lib/bb/build.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 7cefee7..fa297c3 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -169,6 +169,7 @@ def exec_func(func, d, dirs = None):
     cleandirs = flags.get('cleandirs')
     if cleandirs:
         for cdir in d.expand(cleandirs).split():
+            bb.note("removing %s (for %s)" % (cdir, func))
             bb.utils.remove(cdir, True)
             bb.utils.mkdirhier(cdir)
 
@@ -179,6 +180,7 @@ def exec_func(func, d, dirs = None):
 
     if dirs:
         for adir in dirs:
+            bb.note("creating explicit %s (for %s)" % (adir, func))
             bb.utils.mkdirhier(adir)
         adir = dirs[-1]
     else:
@@ -251,7 +253,13 @@ def exec_func_python(func, d, runfile, cwd=None):
             olddir = os.getcwd()
         except OSError:
             olddir = None
+        bb.note("chdir(%s) for %s" % (cwd, func))
         os.chdir(cwd)
+    else:
+        try:
+            bb.note("pwd(%s) for %s" % (os.getcwd(), func))
+        except:
+            bb.note("no chdir for %s" % func)
 
     bb.debug(2, "Executing python function %s" % func)
 
@@ -301,6 +309,13 @@ def exec_func_shell(func, d, runfile, cwd=None):
     # Don't let the emitted shell script override PWD
     d.delVarFlag('PWD', 'export')
 
+    if cwd:
+        bb.note("shell for %s cd %s" % (func, cwd))
+    else:
+        try:
+            bb.note("pwd(%s) for %s" % (os.getcwd(), func))
+        except:
+            bb.note("no cd for %s" % func)
     with open(runfile, 'w') as script:
         script.write(shell_trap_code())
 
-- 
2.1.4

[PATCH 3/7] ptest: add dirs

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/classes/ptest.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/ptest.bbclass b/meta/classes/ptest.bbclass
index 101ab9d..2ac9143 100644
--- a/meta/classes/ptest.bbclass
+++ b/meta/classes/ptest.bbclass
@@ -46,6 +46,9 @@ do_install_ptest_base() {
     fi
 }
 
+do_configure_ptest_base[dirs] = "${B}"
+do_compile_ptest_base[dirs] = "${B}"
+do_install_ptest_base[dirs] = "${B}"
 do_install_ptest_base[cleandirs] = "${D}${PTEST_PATH}"
 
 addtask configure_ptest_base after do_configure before do_compile
-- 
2.1.4

[PATCH 4/7] boost: add dirs

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/recipes-support/boost/boost.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-support/boost/boost.inc b/meta/recipes-support/boost/boost.inc
index cf4faca..27d489f 100644
--- a/meta/recipes-support/boost/boost.inc
+++ b/meta/recipes-support/boost/boost.inc
@@ -154,6 +154,7 @@ do_boostconfig() {
 	sed -i '/^using python/d' project-config.jam
 }
 
+do_boostconfig[dirs] = "${S}"
 addtask do_boostconfig after do_patch before do_configure
 
 do_compile() {
-- 
2.1.4

[PATCH 5/7] python: add explicit directory when copying

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/recipes-devtools/python/python3_3.4.3.bb | 2 +-
 meta/recipes-devtools/python/python_2.7.9.bb  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/python/python3_3.4.3.bb b/meta/recipes-devtools/python/python3_3.4.3.bb
index 3efa7de..dce778a 100644
--- a/meta/recipes-devtools/python/python3_3.4.3.bb
+++ b/meta/recipes-devtools/python/python3_3.4.3.bb
@@ -180,7 +180,7 @@ PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
 
 py_package_preprocess () {
 	# copy back the old Makefile to fix target package
-	install -m 0644 Makefile.orig ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
+	install -m 0644 ${B}/Makefile.orig ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
 	# Remove references to buildmachine paths in target Makefile
 	sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
 }
diff --git a/meta/recipes-devtools/python/python_2.7.9.bb b/meta/recipes-devtools/python/python_2.7.9.bb
index 19190c3..803a1fc 100644
--- a/meta/recipes-devtools/python/python_2.7.9.bb
+++ b/meta/recipes-devtools/python/python_2.7.9.bb
@@ -135,7 +135,7 @@ PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
 
 py_package_preprocess () {
 	# copy back the old Makefile to fix target package
-	install -m 0644 Makefile.orig ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
+	install -m 0644 ${B}/Makefile.orig ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
 
 	# Remove references to buildmachine paths in target Makefile
 	sed -i -e 's:--sysroot=${STAGING_DIR_TARGET}::g' -e s:'--with-libtool-sysroot=${STAGING_DIR_TARGET}'::g ${PKGD}/${libdir}/python${PYTHON_MAJMIN}/config/Makefile
-- 
2.1.4

[PATCH 6/7] libc-package: fix path assumption

[Ross Burton]

Relying on exec_func() doing a mkdir and chdir if you set ${B} is obscure,
instead just pass -C <dir> to oe_runmake.

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/classes/libc-package.bbclass | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/libc-package.bbclass b/meta/classes/libc-package.bbclass
index 2076aa1..21c6b43 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -363,10 +363,9 @@ python package_do_split_gconvs () {
             m.write(cmd + ":\n")
             m.write("\t" + commands[cmd] + "\n\n")
         m.close()
-        d.setVar("B", os.path.dirname(makefile))
         d.setVar("EXTRA_OEMAKE", "${PARALLEL_MAKE}")
         bb.note("Executing binary locale generation makefile")
-        bb.build.exec_func("oe_runmake", d)
+        bb.build.exec_func("oe_runmake", d, [os.path.dirname(makefile),])
         bb.note("collecting binary locales from locale tree")
         bb.build.exec_func("do_collect_bins_from_locale_tree", d)
         do_split_packages(d, binary_locales_dir, file_regex='(.*)', \
-- 
2.1.4

[PATCH 7/7] kernel: fix path assumptions

[Ross Burton]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/classes/kernel.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 76d03dc..d06f6cf 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -211,8 +211,8 @@ kernel_do_compile() {
 
 do_compile_kernelmodules() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
-	if (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
-		oe_runmake ${PARALLEL_MAKE} modules CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS}
+	if (grep -q -i -e '^CONFIG_MODULES=y$' ${B}/.config); then
+		oe_runmake -C ${B} ${PARALLEL_MAKE} modules CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS}
 	else
 		bbnote "no modules to compile"
 	fi
@@ -260,7 +260,7 @@ emit_depmod_pkgdata() {
 	# Stash data for depmod
 	install -d ${PKGDESTWORK}/kernel-depmod/
 	echo "${KERNEL_VERSION}" > ${PKGDESTWORK}/kernel-depmod/kernel-abiversion
-	cp System.map ${PKGDESTWORK}/kernel-depmod/System.map-${KERNEL_VERSION}
+	cp ${B}/System.map ${PKGDESTWORK}/kernel-depmod/System.map-${KERNEL_VERSION}
 }
 
 PACKAGEFUNCS += "emit_depmod_pkgdata"
@@ -336,7 +336,7 @@ kernel_do_configure() {
 }
 
 do_savedefconfig() {
-	oe_runmake savedefconfig
+	oe_runmake -C ${B} savedefconfig
 }
 do_savedefconfig[nostamp] = "1"
 addtask savedefconfig after do_configure
-- 
2.1.4

Re: [PATCH 6/7] libc-package: fix path assumption

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1531 bytes --]

On Mon, Jul 13, 2015 at 7:03 AM, Ross Burton <ross.burton@intel.com> wrote:

> Relying on exec_func() doing a mkdir and chdir if you set ${B} is obscure,
> instead just pass -C <dir> to oe_runmake.
>
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
>  meta/classes/libc-package.bbclass | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/meta/classes/libc-package.bbclass
> b/meta/classes/libc-package.bbclass
> index 2076aa1..21c6b43 100644
> --- a/meta/classes/libc-package.bbclass
> +++ b/meta/classes/libc-package.bbclass
> @@ -363,10 +363,9 @@ python package_do_split_gconvs () {
>              m.write(cmd + ":\n")
>              m.write("\t" + commands[cmd] + "\n\n")
>          m.close()
> -        d.setVar("B", os.path.dirname(makefile))
>          d.setVar("EXTRA_OEMAKE", "${PARALLEL_MAKE}")
>          bb.note("Executing binary locale generation makefile")
> -        bb.build.exec_func("oe_runmake", d)
> +        bb.build.exec_func("oe_runmake", d, [os.path.dirname(makefile),])
>          bb.note("collecting binary locales from locale tree")
>          bb.build.exec_func("do_collect_bins_from_locale_tree", d)
>          do_split_packages(d, binary_locales_dir, file_regex='(.*)', \
>

This says you’re passing -C <dir> to oe_runmake, but actually passes the
dir as dirs for exec_func.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2203 bytes --]

Re: [PATCH 6/7] libc-package: fix path assumption

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On 13 July 2015 at 15:38, Christopher Larson <clarson@kergoth.com> wrote:

> This says you’re passing -C <dir> to oe_runmake, but actually passes the
> dir as dirs for exec_func.
>

Good point - passing -C doesn't work as I was being stupid (bitbake can't
find a function called "oe_runmake -C ${B}").  Fixing comment in the repo.

Ross

[-- Attachment #2: Type: text/html, Size: 821 bytes --]

Re: [PATCH 0/7][RFC][HEREBEDRAGONS] Remove bitbake's knowledge of ${B}

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 630 bytes --]

On 13 July 2015 at 15:02, Ross Burton <ross.burton@intel.com> wrote:

> Ross Burton (7):
>       build: don't automatically mkdir/chdir B
>       build: add remove/mkdir/chdir debugging
>       ptest: add dirs
>       boost: add dirs
>       python: add explicit directory when copying
>       libc-package: fix path assumption
>       kernel: fix path assumptions
>

FYI, I've fixed the libc-package fix (to pass -C to EXTRA_OEMAKE),
elaborated on the commit messages, and merged the oe-core parts into my
master-under-test branch.  The actual change to bitbake is still not merged
and open for comments.

Ross

[-- Attachment #2: Type: text/html, Size: 1075 bytes --]