[PATCH 0/3] Poky environment tweaks + imagetest change

[PATCH 0/3] Poky environment tweaks + imagetest change

[Richard Purdie]

This patch set contains an RFC for some changes to the Poky environment 
handing. These are to allow us to minimise the nunmber of variables our
standard tasks see and hence allow improvements to sstate to be made.
I think the changes are very positive but they need further testing before
they can go into master. In particular this removes variables like DISPLAY
and DBUS socket addresses and so forth, only making them available to tasks
that specifically require them. Some changes to interactive patch reject 
handling will probably be needed to complete this change.

The imagetest change I'd appreciate feed back from Scott Garman
and Jiajun on.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: rpurdie/tweaks2
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2

Cheers,

Richard
---


Richard Purdie (3):
  imagetest-qemu: Allow the task to run after any rootfs is created and
    also standalone
  bitbake: Overhaul environment handling
  devshell: Clean up devshell code so this task doesn't polute the
    global namespace

 bitbake/bin/bitbake-runtask         |    3 ++
 bitbake/lib/bb/cooker.py            |    3 +-
 bitbake/lib/bb/data.py              |   10 ++++++-
 bitbake/lib/bb/runqueue.py          |    1 +
 bitbake/lib/bb/utils.py             |   48 ++++++++++++++++++++++++-----------
 meta/classes/devshell.bbclass       |   11 +++----
 meta/classes/imagetest-qemu.bbclass |   19 ++++++++++---
 7 files changed, 67 insertions(+), 28 deletions(-)

[PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone

[Richard Purdie]

One bitbake invocation can cause multiple images to be generated. We should test
each one and we cna do this by running after the rootfs task. Running the tests
standlone is also still possible with a new separate standlone task.

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
---
 meta/classes/imagetest-qemu.bbclass |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/meta/classes/imagetest-qemu.bbclass b/meta/classes/imagetest-qemu.bbclass
index 07bdf01..3c1586e 100644
--- a/meta/classes/imagetest-qemu.bbclass
+++ b/meta/classes/imagetest-qemu.bbclass
@@ -1,8 +1,3 @@
-addtask qemuimagetest before do_build
-# after do_rootfs
-do_qemuimagetest[nostamp] = "1"
-do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
-
 # Test related variables
 # By default, TEST_DIR is created under WORKDIR
 TEST_DIR ?= "${WORKDIR}/qemuimagetest"
@@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp"
 TEST_SCEN ?= "sanity"
 
 python do_qemuimagetest() {
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest before do_build after do_rootfs
+do_qemuimagetest[nostamp] = "1"
+do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
+
+python do_qemuimagetest_standalone() {
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest_standalone
+do_qemuimagetest_standalone[nostamp] = "1"
+do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot"
+
+python qemuimagetest_main(d) {
     import sys
     import re
     import os
-- 
1.7.0.4

Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone

[Xu, Jiajun]

> One bitbake invocation can cause multiple images to be generated. We
> should test each one and we cna do this by running after the rootfs
> task. Running the tests standlone is also still possible with a new separate standlone task.
> 
> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
> ---
>  meta/classes/imagetest-qemu.bbclass |   19 ++++++++++++++-----
>  1 files changed, 14 insertions(+), 5 deletions(-)
> diff --git a/meta/classes/imagetest-qemu.bbclass
> b/meta/classes/imagetest-qemu.bbclass index 07bdf01..3c1586e 100644 ---
> a/meta/classes/imagetest-qemu.bbclass +++
> b/meta/classes/imagetest-qemu.bbclass @@ -1,8 +1,3 @@ -addtask
> qemuimagetest before do_build -# after do_rootfs
> -do_qemuimagetest[nostamp] = "1" -do_qemuimagetest[depends] +=
> "qemu-native:do_populate_sysroot" -
>  # Test related variables
>  # By default, TEST_DIR is created under WORKDIR  TEST_DIR ?=
> "${WORKDIR}/qemuimagetest"
> @@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp"
>  TEST_SCEN ?= "sanity"
>  
>  python do_qemuimagetest() {
> +    qemuimagetest_main(d) +} +addtask qemuimagetest before do_build
> after do_rootfs +do_qemuimagetest[nostamp] = "1"
> +do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" +
> +python do_qemuimagetest_standalone() { +    qemuimagetest_main(d) +}
> +addtask qemuimagetest_standalone +do_qemuimagetest_standalone[nostamp]
> = "1" +do_qemuimagetest_standalone[depends] +=
> "qemu-native:do_populate_sysroot" + +python qemuimagetest_main(d) {
>      import sys
>      import re
>      import os

Hi Richard,
Seems we should use keyword "def" qemuimagetest_main definition. After this modification, it can work on my side.

diff --git a/meta/classes/imagetest-qemu.bbclass b/meta/classes/imagetest-qemu.bbclass
index 07bdf01..8751eeb 100644
--- a/meta/classes/imagetest-qemu.bbclass
+++ b/meta/classes/imagetest-qemu.bbclass
@@ -1,8 +1,3 @@
-addtask qemuimagetest before do_build
-# after do_rootfs
-do_qemuimagetest[nostamp] = "1"
-do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
-
 # Test related variables
 # By default, TEST_DIR is created under WORKDIR
 TEST_DIR ?= "${WORKDIR}/qemuimagetest"
@@ -11,7 +6,21 @@ TEST_RESULT ?= "${TEST_DIR}/result"
 TEST_TMP ?= "${TEST_DIR}/tmp"
 TEST_SCEN ?= "sanity"

-python do_qemuimagetest() {
+python do_qemuimagetest(){
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest before do_build after do_rootfs
+do_qemuimagetest[nostamp] = "1"
+do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
+
+python do_qemuimagetest_standalone(){
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest_standalone
+do_qemuimagetest_standalone[nostamp] = "1"
+do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot"
+
+def qemuimagetest_main(d):
     import sys
     import re
     import os
@@ -142,4 +151,3 @@ python do_qemuimagetest() {

     if ret != 0:
         raise bb.build.FuncFailed("Some testcases fail, pls. check test result and test log!!!")
-}


Best Regards,
Jiajun

Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone

[Scott Garman]

On 11/28/2010 06:45 AM, Richard Purdie wrote:
> One bitbake invocation can cause multiple images to be generated. We should test
> each one and we cna do this by running after the rootfs task. Running the tests
> standlone is also still possible with a new separate standlone task.
>
> Signed-off-by: Richard Purdie<rpurdie@linux.intel.com>

Acked-by: Scott Garman <scott.a.garman@intel.com>

(with Jiajun's modification)

> ---
>   meta/classes/imagetest-qemu.bbclass |   19 ++++++++++++++-----
>   1 files changed, 14 insertions(+), 5 deletions(-)
>
> diff --git a/meta/classes/imagetest-qemu.bbclass b/meta/classes/imagetest-qemu.bbclass
> index 07bdf01..3c1586e 100644
> --- a/meta/classes/imagetest-qemu.bbclass
> +++ b/meta/classes/imagetest-qemu.bbclass
> @@ -1,8 +1,3 @@
> -addtask qemuimagetest before do_build
> -# after do_rootfs
> -do_qemuimagetest[nostamp] = "1"
> -do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
> -
>   # Test related variables
>   # By default, TEST_DIR is created under WORKDIR
>   TEST_DIR ?= "${WORKDIR}/qemuimagetest"
> @@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp"
>   TEST_SCEN ?= "sanity"
>
>   python do_qemuimagetest() {
> +    qemuimagetest_main(d)
> +}
> +addtask qemuimagetest before do_build after do_rootfs
> +do_qemuimagetest[nostamp] = "1"
> +do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
> +
> +python do_qemuimagetest_standalone() {
> +    qemuimagetest_main(d)
> +}
> +addtask qemuimagetest_standalone
> +do_qemuimagetest_standalone[nostamp] = "1"
> +do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot"
> +
> +python qemuimagetest_main(d) {
>       import sys
>       import re
>       import os


-- 
Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone

[Richard Purdie]

On Fri, 2010-12-03 at 20:58 -0800, Scott Garman wrote:
> On 11/28/2010 06:45 AM, Richard Purdie wrote:
> > One bitbake invocation can cause multiple images to be generated. We should test
> > each one and we cna do this by running after the rootfs task. Running the tests
> > standlone is also still possible with a new separate standlone task.
> >
> > Signed-off-by: Richard Purdie<rpurdie@linux.intel.com>
> 
> Acked-by: Scott Garman <scott.a.garman@intel.com>
> 
> (with Jiajun's modification)

Thanks, I've merged this into master with the fix.

Cheers,

Richard

[PATCH 2/3] bitbake: Overhaul environment handling

[Richard Purdie]

Currently, anything whitelisted in the environment makes it into the worker
processes. This is undesireable and the worker environment should be as
clean as possible. This patch adapts bitbake sosme variables are loaded into
bitbake's datastore but not exported by default. Any variable can be exported
by setting its export flag.

Currently, this code only finalises the environment in he worker as doing so
in the server means variables are unavailable in the worker. If we switch
back to fork() calls instead of exec() this code will need revisting.

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
---
 bitbake/bin/bitbake-runtask |    3 ++
 bitbake/lib/bb/cooker.py    |    3 +-
 bitbake/lib/bb/data.py      |   10 ++++++++-
 bitbake/lib/bb/runqueue.py  |    1 +
 bitbake/lib/bb/utils.py     |   48 +++++++++++++++++++++++++++++-------------
 5 files changed, 48 insertions(+), 17 deletions(-)

diff --git a/bitbake/bin/bitbake-runtask b/bitbake/bin/bitbake-runtask
index 88101a5..9079f57 100755
--- a/bitbake/bin/bitbake-runtask
+++ b/bitbake/bin/bitbake-runtask
@@ -100,6 +100,9 @@ the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.
 cooker.bb_cache.setData(fn, buildfile, the_data)
 cooker.bb_cache.handle_data(fn, cooker.status)
 
+exportlist = bb.utils.preserved_envvars_export_list()
+bb.utils.filter_environment(exportlist)
+
 if taskname.endswith("_setscene"):
     the_data.setVarFlag(taskname, "quieterrors", "1")
 
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 054dac8..3cd358c 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -918,7 +918,8 @@ class BBCooker:
     def pre_serve(self):
         # Empty the environment. The environment will be populated as
         # necessary from the data store.
-        bb.utils.empty_environment()
+        #bb.utils.empty_environment()
+        return
 
     def post_serve(self):
         bb.event.fire(CookerExit(), self.configuration.event_data)
diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
index fee10cc..d4d43fd 100644
--- a/bitbake/lib/bb/data.py
+++ b/bitbake/lib/bb/data.py
@@ -161,10 +161,12 @@ def expandKeys(alterdata, readdata = None):
 
 def inheritFromOS(d):
     """Inherit variables from the environment."""
+    exportlist = bb.utils.preserved_envvars_export_list()
     for s in os.environ.keys():
         try:
             setVar(s, os.environ[s], d)
-            setVarFlag(s, "export", True, d)
+            if s in exportlist:
+                setVarFlag(s, "export", True, d)
         except TypeError:
             pass
 
@@ -244,6 +246,12 @@ def export_vars(d):
             pass
     return ret
 
+def export_envvars(v, d):
+    for s in os.environ.keys():
+        if s not in v:
+            v[s] = os.environ[s]
+    return v
+
 def emit_func(func, o=sys.__stdout__, d = init()):
     """Emits all items in the data store in a format such that it can be sourced by a shell."""
 
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 3300db7..e26aa4e 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1062,6 +1062,7 @@ class RunQueueExecute:
             the_data = self.cooker.bb_cache.loadDataFull(fn, self.cooker.get_file_appends(fn), self.cooker.configuration.data)
 
             env = bb.data.export_vars(the_data)
+            env = bb.data.export_envvars(env, the_data)
 
             taskdep = self.rqdata.dataCache.task_deps[fn]
             if 'fakeroot' in taskdep and taskname in taskdep['fakeroot']:
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index c2e6ff0..0c315a4 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -446,13 +446,25 @@ def sha256_file(filename):
         s.update(line)
     return s.hexdigest()
 
-def preserved_envvars_list():
+# Variables which are preserved from the original environment *and* exported
+# into our worker context
+def preserved_envvars_export_list():
     return [
-        'BBPATH',
-        'BB_PRESERVE_ENV',
-        'BB_ENV_WHITELIST',
-        'BB_ENV_EXTRAWHITE',
         'BB_TASKHASH',
+        'HOME',
+        'LOGNAME',
+        'PATH',
+        'PWD',
+        'SHELL',
+        'TERM',
+        'USER',
+        'USERNAME',
+    ]
+
+# Variables which are preserved from the original environment *and* exported
+# into our worker context for interactive tasks (e.g. requiring X)
+def preserved_envvars_export_interactive_list():
+    return [
         'COLORTERM',
         'DBUS_SESSION_BUS_ADDRESS',
         'DESKTOP_SESSION',
@@ -462,23 +474,25 @@ def preserved_envvars_list():
         'GNOME_KEYRING_SOCKET',
         'GPG_AGENT_INFO',
         'GTK_RC_FILES',
-        'HOME',
-        'LANG',
-        'LOGNAME',
-        'PATH',
-        'PWD',
         'SESSION_MANAGER',
-        'SHELL',
         'SSH_AUTH_SOCK',
-        'TERM',
-        'USER',
-        'USERNAME',
-        '_',
         'XAUTHORITY',
         'XDG_DATA_DIRS',
         'XDG_SESSION_COOKIE',
     ]
 
+# Variables which are preserved from the original environment into the datastore
+def preserved_envvars_list():
+    v = [
+        'BBPATH',
+        'BB_PRESERVE_ENV',
+        'BB_ENV_WHITELIST',
+        'BB_ENV_EXTRAWHITE',
+        'LANG',
+        '_',
+    ]
+    return v + preserved_envvars_export_list() + preserved_envvars_export_interactive_list()
+
 def filter_environment(good_vars):
     """
     Create a pristine environment for bitbake. This will remove variables that
@@ -499,6 +513,10 @@ def filter_environment(good_vars):
 
     return removed_vars
 
+def create_intereactive_env(d):
+    for k in preserved_envvars_export_interactive_list():
+        os.setenv(k, bb.data.getVar(k, d, True))
+
 def clean_environment():
     """
     Clean up any spurious environment variables. This will remove any
-- 
1.7.0.4

[PATCH 3/3] devshell: Clean up devshell code so this task doesn't polute the global namespace

[Richard Purdie]

Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
---
 meta/classes/devshell.bbclass |   11 +++++------
 1 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/meta/classes/devshell.bbclass b/meta/classes/devshell.bbclass
index d08056b..4fb3e63 100644
--- a/meta/classes/devshell.bbclass
+++ b/meta/classes/devshell.bbclass
@@ -1,15 +1,14 @@
-EXTRA_OEMAKE[export] = "1"
-
 do_devshell[dirs] = "${S}"
 do_devshell[nostamp] = "1"
 
-export DISPLAY
-export DBUS_SESSION_BUS_ADDRESS
-export XAUTHORITY ?= "${HOME}/.Xauthority"
-export FAKEROOTENV
+XAUTHORITY ?= "${HOME}/.Xauthority"
 
 devshell_do_devshell() {
+	export DISPLAY='${DISPLAY}'
+	export DBUS_SESSION_BUS_ADDRESS='${DBUS_SESSION_BUS_ADDRESS}'
+	export XAUTHORITY='${XAUTHORITY}'
 	export TERMWINDOWTITLE="Bitbake Developer Shell"
+	export EXTRA_OEMAKE='${EXTRA_OEMAKE}'
 	${TERMCMD}
 	if [ $? -ne 0 ]; then
 	    echo "Fatal: '${TERMCMD}' not found. Check TERMCMD variable."
-- 
1.7.0.4