[PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation

[PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation

[Mark Hatle]

It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not 
complete.  The version that was merged back in May only contained the 
filesystem generation pieces, but not the pieces for creating the image
from that filesystem.

The code has been tested and is working.  The only thing that I don't 
particularly like is that the processing code and loop is a duplicate of
the code that runs just before.  Unfortunately the only way around this
is to change the way that way the parallel bits are processed to support
multiple datastores..  (or create "another" function..)

Any feedback appreciated, but without this the feature is broken!

Mark Hatle (1):
  lib/oe/image.py: Add image generation for companion debug filesystem

 meta/conf/documentation.conf         |  2 ++
 meta/conf/local.conf.sample.extended |  9 +++++++++
 meta/lib/oe/image.py                 | 37 ++++++++++++++++++++++++++++++++++++
 3 files changed, 48 insertions(+)

-- 
1.9.3

[PATCH] lib/oe/image.py: Add image generation for companion debug filesystem

[Mark Hatle]

The companion debug filesystem, enabled with IMAGE_GEN_DEBUGFS, was
creating the companion filesystem but was missing the code to actually
package it into a usable filesystem.

The code (and associated documentation) will allow the debugfs to generate a
companion tarball or other image.

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/conf/documentation.conf         |  2 ++
 meta/conf/local.conf.sample.extended |  9 +++++++++
 meta/lib/oe/image.py                 | 37 ++++++++++++++++++++++++++++++++++++
 3 files changed, 48 insertions(+)

diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 075ab6a..845559a 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -212,6 +212,8 @@ IMAGE_BOOT_FILES[doc] = "Whitespace separated list of files from ${DEPLOY_DIR_IM
 IMAGE_CLASSES[doc] = "A list of classes that all images should inherit."
 IMAGE_FEATURES[doc] = "The primary list of features to include in an image. Configure this variable in an image recipe."
 IMAGE_FSTYPES[doc] = "Formats of root filesystem images that you want to have created."
+IMAGE_FSTYPES_DEBUGFS[doc] = "Formats of the debug root filesystem images that you want to have created."
+IMAGE_GEN_DEBUGFS[doc] = "When set to '1', generate a companion debug object/source filesystem image."
 IMAGE_INSTALL[doc] = "Specifies the packages to install into an image. Image recipes set IMAGE_INSTALL to specify the packages to install into an image through image.bbclass."
 IMAGE_LINGUAS[doc] = "Specifies the list of locales to install into the image during the root filesystem construction process."
 IMAGE_NAME[doc] = "The name of the output image files minus the extension."
diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended
index ccdd326..bc765a1 100644
--- a/meta/conf/local.conf.sample.extended
+++ b/meta/conf/local.conf.sample.extended
@@ -165,6 +165,15 @@
 # currently an example class is image_types_uboot
 # IMAGE_CLASSES = " image_types_uboot"
 
+# The following options will build a companion 'debug filesystem' in addition
+# to the normal deployable filesystem.  This companion system allows a
+# debugger to know the symbols and related sources.  It can be used to
+# debug a remote 'production' system without having to add the debug symbols
+# and sources to remote system.  If IMAGE_FSTYPES_DEBUGFS is not defined, it
+# defaults to IMAGE_FSTYPES.
+#IMAGE_GEN_DEBUGFS = "1"
+#IMAGE_FSTYPES_DEBUGFS = "tar.gz"
+
 # Incremental rpm image generation, the rootfs would be totally removed
 # and re-created in the second generation by default, but with
 # INC_RPM_IMAGE_GEN = "1", the rpm based rootfs would be kept, and will
diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
index f9e9bfd..688df9d 100644
--- a/meta/lib/oe/image.py
+++ b/meta/lib/oe/image.py
@@ -373,6 +373,43 @@ class Image(ImageDepGraph):
                 bb.note("Creating symlinks for %s image ..." % image_type)
                 self._create_symlinks(subimages)
 
+        # Process the debug filesystem...
+        if self.d.getVar('IMAGE_GEN_DEBUGFS', True) == "1":
+            bb.note("Processing debugfs image(s) ...")
+            orig_d = self.d
+            debugfs_d = bb.data.createCopy(self.d)
+            self.d = debugfs_d
+
+            self.d.setVar('IMAGE_ROOTFS', orig_d.getVar('IMAGE_ROOTFS', True) + '-dbg')
+            self.d.setVar('IMAGE_NAME', orig_d.getVar('IMAGE_NAME', True) + '-dbg')
+            self.d.setVar('IMAGE_LINK_NAME', orig_d.getVar('IMAGE_LINK_NAME', True) + '-dbg')
+
+            debugfs_image_fstypes = orig_d.getVar('IMAGE_FSTYPES_DEBUGFS', True)
+            if debugfs_image_fstypes:
+                self.d.setVar('IMAGE_FSTYPES', orig_d.getVar('IMAGE_FSTYPES_DEBUGFS', True))
+
+            self._remove_old_symlinks()
+
+            debugfs_image_cmd_groups = self._get_imagecmds()
+
+            for image_cmds in debugfs_image_cmd_groups:
+                # create the images in parallel
+                nproc = multiprocessing.cpu_count()
+                pool = bb.utils.multiprocessingpool(nproc)
+                results = list(pool.imap(generate_image, image_cmds))
+                pool.close()
+                pool.join()
+
+                for result in results:
+                    if result is not None:
+                        bb.fatal(result)
+
+                for image_type, subimages, script in image_cmds:
+                    bb.note("Creating debugfs symlinks for %s image ..." % image_type)
+                    self._create_symlinks(subimages)
+
+            self.d = orig_d
+
         execute_pre_post_process(self.d, post_process_cmds)
 
 
-- 
1.9.3

Re: [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation

[Richard Purdie]

On Thu, 2015-10-01 at 13:26 -0500, Mark Hatle wrote:
> It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not 
> complete.  The version that was merged back in May only contained the 
> filesystem generation pieces, but not the pieces for creating the image
> from that filesystem.
> 
> The code has been tested and is working.  The only thing that I don't 
> particularly like is that the processing code and loop is a duplicate of
> the code that runs just before.  Unfortunately the only way around this
> is to change the way that way the parallel bits are processed to support
> multiple datastores..  (or create "another" function..)
> 
> Any feedback appreciated, but without this the feature is broken!

Could we not make a function which these two code points then call?

Cheers,

Richard

Re: [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation

[Mark Hatle]

On 10/1/15 5:31 PM, Richard Purdie wrote:
> On Thu, 2015-10-01 at 13:26 -0500, Mark Hatle wrote:
>> It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not 
>> complete.  The version that was merged back in May only contained the 
>> filesystem generation pieces, but not the pieces for creating the image
>> from that filesystem.
>>
>> The code has been tested and is working.  The only thing that I don't 
>> particularly like is that the processing code and loop is a duplicate of
>> the code that runs just before.  Unfortunately the only way around this
>> is to change the way that way the parallel bits are processed to support
>> multiple datastores..  (or create "another" function..)
>>
>> Any feedback appreciated, but without this the feature is broken!
> 
> Could we not make a function which these two code points then call?

The duplicate piece is because the existing setup and loop depend on the local
self.d value(s).  In order to do this, we need to temporarily modify self.d and
run this under an alternative datastore, and then put it back to the original value.

If I don't duplicate the:

            for image_cmds in debugfs_image_cmd_groups:
...
                results = list(pool.imap(generate_image, image_cmds))
...
                for image_type, subimages, script in image_cmds:
                    bb.note("Creating debugfs symlinks for %s image ..." %
image_type)
                    self._create_symlinks(subimages)

there is no concept of two different datastores.

The alternative we have is to include a reference to the datastore itself in the
'image_cmds'.  Then we could support any number of datastores as appropriate for
the commands.  (This of course will require additional changes to be able to
pass that datastore to the various users.)

--Mark



> Cheers,
> 
> Richard
> 

Re: [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation

[Richard Purdie]

On Fri, 2015-10-02 at 09:23 -0500, Mark Hatle wrote:
> On 10/1/15 5:31 PM, Richard Purdie wrote:
> > On Thu, 2015-10-01 at 13:26 -0500, Mark Hatle wrote:
> >> It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not 
> >> complete.  The version that was merged back in May only contained the 
> >> filesystem generation pieces, but not the pieces for creating the image
> >> from that filesystem.
> >>
> >> The code has been tested and is working.  The only thing that I don't 
> >> particularly like is that the processing code and loop is a duplicate of
> >> the code that runs just before.  Unfortunately the only way around this
> >> is to change the way that way the parallel bits are processed to support
> >> multiple datastores..  (or create "another" function..)
> >>
> >> Any feedback appreciated, but without this the feature is broken!
> > 
> > Could we not make a function which these two code points then call?
> 
> The duplicate piece is because the existing setup and loop depend on the local
> self.d value(s).  In order to do this, we need to temporarily modify self.d and
> run this under an alternative datastore, and then put it back to the original value.
> 
> If I don't duplicate the:
> 
>             for image_cmds in debugfs_image_cmd_groups:
> ...
>                 results = list(pool.imap(generate_image, image_cmds))
> ...
>                 for image_type, subimages, script in image_cmds:
>                     bb.note("Creating debugfs symlinks for %s image ..." %
> image_type)
>                     self._create_symlinks(subimages)
> 
> there is no concept of two different datastores.
> 
> The alternative we have is to include a reference to the datastore itself in the
> 'image_cmds'.  Then we could support any number of datastores as appropriate for
> the commands.  (This of course will require additional changes to be able to
> pass that datastore to the various users.)

Ah, right. Going from memory, we can't share a datastore with a
multiprocessing pool. We could do that with a multithreaded pool but I'm
nervous about that for two reasons, one that we've had bad interaction
between threads and processes in the past and secondly around locking
and data usage in the threads.

So not an easy problem...

Cheers,

Richard

Re: [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation

[Mark Hatle]

On 10/2/15 9:30 AM, Richard Purdie wrote:
> On Fri, 2015-10-02 at 09:23 -0500, Mark Hatle wrote:
>> On 10/1/15 5:31 PM, Richard Purdie wrote:
>>> On Thu, 2015-10-01 at 13:26 -0500, Mark Hatle wrote:
>>>> It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not 
>>>> complete.  The version that was merged back in May only contained the 
>>>> filesystem generation pieces, but not the pieces for creating the image
>>>> from that filesystem.
>>>>
>>>> The code has been tested and is working.  The only thing that I don't 
>>>> particularly like is that the processing code and loop is a duplicate of
>>>> the code that runs just before.  Unfortunately the only way around this
>>>> is to change the way that way the parallel bits are processed to support
>>>> multiple datastores..  (or create "another" function..)
>>>>
>>>> Any feedback appreciated, but without this the feature is broken!
>>>
>>> Could we not make a function which these two code points then call?
>>
>> The duplicate piece is because the existing setup and loop depend on the local
>> self.d value(s).  In order to do this, we need to temporarily modify self.d and
>> run this under an alternative datastore, and then put it back to the original value.
>>
>> If I don't duplicate the:
>>
>>             for image_cmds in debugfs_image_cmd_groups:
>> ...
>>                 results = list(pool.imap(generate_image, image_cmds))
>> ...
>>                 for image_type, subimages, script in image_cmds:
>>                     bb.note("Creating debugfs symlinks for %s image ..." %
>> image_type)
>>                     self._create_symlinks(subimages)
>>
>> there is no concept of two different datastores.
>>
>> The alternative we have is to include a reference to the datastore itself in the
>> 'image_cmds'.  Then we could support any number of datastores as appropriate for
>> the commands.  (This of course will require additional changes to be able to
>> pass that datastore to the various users.)
> 
> Ah, right. Going from memory, we can't share a datastore with a
> multiprocessing pool. We could do that with a multithreaded pool but I'm
> nervous about that for two reasons, one that we've had bad interaction
> between threads and processes in the past and secondly around locking
> and data usage in the threads.
> 
> So not an easy problem...

V2 coming..  it turns out you can avoid the multithread pool issue by making a
few other select changes..

I missed this yesterday when I was floundering trying to get this to work.

--Mark

> Cheers,
> 
> Richard
>