[PATCH v2] "Finish" the IMAGE_GEN_DEBUGFS implementation

[PATCH v2] "Finish" the IMAGE_GEN_DEBUGFS implementation

[Mark Hatle]

V2:
By adjusting the _write_script function (and associated _get_imagecmds) the
script may optionally pass in a prefix for the generated script.  This solves
one of the problems in the prior version.  (No more script name collisions.)

The second issue is that the datastore is used to find various parameters
(which are different in the DEBUGFS datastore) when doing the final symlink
generation.

So we do have to switch (based on the script prefix) which datastore we
should be using.  (Note, I'm not completely sure that the switch is 'safe',
so this is an area that should be verified.)

V1:

It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not 
complete.  The version that was merged back in May only contained the 
filesystem generation pieces, but not the pieces for creating the image
from that filesystem.

The code has been tested and is working.  The only thing that I don't 
particularly like is that the processing code and loop is a duplicate of
the code that runs just before.  Unfortunately the only way around this
is to change the way that way the parallel bits are processed to support
multiple datastores..  (or create "another" function..)

Any feedback appreciated, but without this the feature is broken!

Mark Hatle (1):
  lib/oe/image.py: Add image generation for companion debug filesystem

 meta/conf/documentation.conf         |  2 ++
 meta/conf/local.conf.sample.extended |  9 +++++++++
 meta/lib/oe/image.py                 | 37 +++++++++++++++++++++++++++++++-----
 3 files changed, 43 insertions(+), 5 deletions(-)

-- 
1.9.3

[PATCH v2] lib/oe/image.py: Add image generation for companion debug filesystem

[Mark Hatle]

The companion debug filesystem, enabled with IMAGE_GEN_DEBUGFS, was
creating the companion filesystem but was missing the code to actually
package it into a usable filesystem.

The code (and associated documentation) will allow the debugfs to generate a
companion tarball or other image.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/conf/documentation.conf         |  2 ++
 meta/conf/local.conf.sample.extended |  9 +++++++++
 meta/lib/oe/image.py                 | 37 +++++++++++++++++++++++++++++++-----
 3 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 075ab6a..845559a 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -212,6 +212,8 @@ IMAGE_BOOT_FILES[doc] = "Whitespace separated list of files from ${DEPLOY_DIR_IM
 IMAGE_CLASSES[doc] = "A list of classes that all images should inherit."
 IMAGE_FEATURES[doc] = "The primary list of features to include in an image. Configure this variable in an image recipe."
 IMAGE_FSTYPES[doc] = "Formats of root filesystem images that you want to have created."
+IMAGE_FSTYPES_DEBUGFS[doc] = "Formats of the debug root filesystem images that you want to have created."
+IMAGE_GEN_DEBUGFS[doc] = "When set to '1', generate a companion debug object/source filesystem image."
 IMAGE_INSTALL[doc] = "Specifies the packages to install into an image. Image recipes set IMAGE_INSTALL to specify the packages to install into an image through image.bbclass."
 IMAGE_LINGUAS[doc] = "Specifies the list of locales to install into the image during the root filesystem construction process."
 IMAGE_NAME[doc] = "The name of the output image files minus the extension."
diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
index ccdd326..bc765a1 100644
--- a/meta/conf/local.conf.sample.extended
+++ b/meta/conf/local.conf.sample.extended
@@ -165,6 +165,15 @@
 # currently an example class is image_types_uboot
 # IMAGE_CLASSES = " image_types_uboot"
 
+# The following options will build a companion 'debug filesystem' in addition
+# to the normal deployable filesystem.  This companion system allows a
+# debugger to know the symbols and related sources.  It can be used to
+# debug a remote 'production' system without having to add the debug symbols
+# and sources to remote system.  If IMAGE_FSTYPES_DEBUGFS is not defined, it
+# defaults to IMAGE_FSTYPES.
+#IMAGE_GEN_DEBUGFS = "1"
+#IMAGE_FSTYPES_DEBUGFS = "tar.gz"
+
 # Incremental rpm image generation, the rootfs would be totally removed
 # and re-created in the second generation by default, but with
 # INC_RPM_IMAGE_GEN = "1", the rpm based rootfs would be kept, and will
diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
index f9e9bfd..8657539 100644
--- a/meta/lib/oe/image.py
+++ b/meta/lib/oe/image.py
@@ -264,9 +264,9 @@ class Image(ImageDepGraph):
 
         return (alltypes, filtered_groups, cimages)
 
-    def _write_script(self, type, cmds):
+    def _write_script(self, type, cmds, sprefix=""):
         tempdir = self.d.getVar('T', True)
-        script_name = os.path.join(tempdir, "create_image." + type)
+        script_name = os.path.join(tempdir, sprefix + "create_image." + type)
         rootfs_size = self._get_rootfs_size()
 
         self.d.setVar('img_creation_func', '\n'.join(cmds))
@@ -284,7 +284,7 @@ class Image(ImageDepGraph):
 
         return script_name
 
-    def _get_imagecmds(self):
+    def _get_imagecmds(self, sprefix=""):
         old_overrides = self.d.getVar('OVERRIDES', 0)
 
         alltypes, fstype_groups, cimages = self._get_image_types()
@@ -320,7 +320,7 @@ class Image(ImageDepGraph):
                 else:
                     subimages.append(type)
 
-                script_name = self._write_script(type, cmds)
+                script_name = self._write_script(type, cmds, sprefix)
 
                 image_cmds.append((type, subimages, script_name))
 
@@ -355,6 +355,27 @@ class Image(ImageDepGraph):
 
         image_cmd_groups = self._get_imagecmds()
 
+        # Process the debug filesystem...
+        debugfs_d = bb.data.createCopy(self.d)
+        if self.d.getVar('IMAGE_GEN_DEBUGFS', True) == "1":
+            bb.note("Processing debugfs image(s) ...")
+            orig_d = self.d
+            self.d = debugfs_d
+
+            self.d.setVar('IMAGE_ROOTFS', orig_d.getVar('IMAGE_ROOTFS', True) + '-dbg')
+            self.d.setVar('IMAGE_NAME', orig_d.getVar('IMAGE_NAME', True) + '-dbg')
+            self.d.setVar('IMAGE_LINK_NAME', orig_d.getVar('IMAGE_LINK_NAME', True) + '-dbg')
+
+            debugfs_image_fstypes = orig_d.getVar('IMAGE_FSTYPES_DEBUGFS', True)
+            if debugfs_image_fstypes:
+                self.d.setVar('IMAGE_FSTYPES', orig_d.getVar('IMAGE_FSTYPES_DEBUGFS', True))
+
+            self._remove_old_symlinks()
+
+            image_cmd_groups += self._get_imagecmds("debugfs.")
+
+            self.d = orig_d
+
         self._write_wic_env()
 
         for image_cmds in image_cmd_groups:
@@ -371,7 +392,13 @@ class Image(ImageDepGraph):
 
             for image_type, subimages, script in image_cmds:
                 bb.note("Creating symlinks for %s image ..." % image_type)
-                self._create_symlinks(subimages)
+                if 'debugfs.' in script:
+                    orig_d = self.d
+                    self.d = debugfs_d
+                    self._create_symlinks(subimages)
+                    self.d = orig_d
+                else:
+                    self._create_symlinks(subimages)
 
         execute_pre_post_process(self.d, post_process_cmds)
 
-- 
1.9.3

Re: [PATCH v2] "Finish" the IMAGE_GEN_DEBUGFS implementation

[Mark Hatle]

On 10/2/15 9:53 AM, Mark Hatle wrote:
> V2:
> By adjusting the _write_script function (and associated _get_imagecmds) the
> script may optionally pass in a prefix for the generated script.  This solves
> one of the problems in the prior version.  (No more script name collisions.)
> 
> The second issue is that the datastore is used to find various parameters
> (which are different in the DEBUGFS datastore) when doing the final symlink
> generation.
> 
> So we do have to switch (based on the script prefix) which datastore we
> should be using.  (Note, I'm not completely sure that the switch is 'safe',
> so this is an area that should be verified.)

And I verified, it is indeed not safe..

The script name is set by:

        script_name = os.path.join(tempdir, sprefix + "create_image." + type)

So I'd need to exclude the tempdir ($T) in order for the test to be right.. and
even then it probably should be a 'startwith' comparison?

Does this seem right to others?

--Mark

> V1:
> 
> It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not 
> complete.  The version that was merged back in May only contained the 
> filesystem generation pieces, but not the pieces for creating the image
> from that filesystem.
> 
> The code has been tested and is working.  The only thing that I don't 
> particularly like is that the processing code and loop is a duplicate of
> the code that runs just before.  Unfortunately the only way around this
> is to change the way that way the parallel bits are processed to support
> multiple datastores..  (or create "another" function..)
> 
> Any feedback appreciated, but without this the feature is broken!
> 
> Mark Hatle (1):
>   lib/oe/image.py: Add image generation for companion debug filesystem
> 
>  meta/conf/documentation.conf         |  2 ++
>  meta/conf/local.conf.sample.extended |  9 +++++++++
>  meta/lib/oe/image.py                 | 37 +++++++++++++++++++++++++++++++-----
>  3 files changed, 43 insertions(+), 5 deletions(-)
>