* [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation @ 2015-10-01 18:26 Mark Hatle 2015-10-01 18:26 ` [PATCH] lib/oe/image.py: Add image generation for companion debug filesystem Mark Hatle 2015-10-01 22:31 ` [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation Richard Purdie 0 siblings, 2 replies; 6+ messages in thread From: Mark Hatle @ 2015-10-01 18:26 UTC (permalink / raw) To: openembedded-core; +Cc: david.reyna It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not complete. The version that was merged back in May only contained the filesystem generation pieces, but not the pieces for creating the image from that filesystem. The code has been tested and is working. The only thing that I don't particularly like is that the processing code and loop is a duplicate of the code that runs just before. Unfortunately the only way around this is to change the way that way the parallel bits are processed to support multiple datastores.. (or create "another" function..) Any feedback appreciated, but without this the feature is broken! Mark Hatle (1): lib/oe/image.py: Add image generation for companion debug filesystem meta/conf/documentation.conf | 2 ++ meta/conf/local.conf.sample.extended | 9 +++++++++ meta/lib/oe/image.py | 37 ++++++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+) -- 1.9.3 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] lib/oe/image.py: Add image generation for companion debug filesystem 2015-10-01 18:26 [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation Mark Hatle @ 2015-10-01 18:26 ` Mark Hatle 2015-10-01 22:31 ` [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation Richard Purdie 1 sibling, 0 replies; 6+ messages in thread From: Mark Hatle @ 2015-10-01 18:26 UTC (permalink / raw) To: openembedded-core; +Cc: david.reyna The companion debug filesystem, enabled with IMAGE_GEN_DEBUGFS, was creating the companion filesystem but was missing the code to actually package it into a usable filesystem. The code (and associated documentation) will allow the debugfs to generate a companion tarball or other image. Signed-off-by: Mark Hatle <mark.hatle@windriver.com> --- meta/conf/documentation.conf | 2 ++ meta/conf/local.conf.sample.extended | 9 +++++++++ meta/lib/oe/image.py | 37 ++++++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+) diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf index 075ab6a..845559a 100644 --- a/meta/conf/documentation.conf +++ b/meta/conf/documentation.conf @@ -212,6 +212,8 @@ IMAGE_BOOT_FILES[doc] = "Whitespace separated list of files from ${DEPLOY_DIR_IM IMAGE_CLASSES[doc] = "A list of classes that all images should inherit." IMAGE_FEATURES[doc] = "The primary list of features to include in an image. Configure this variable in an image recipe." IMAGE_FSTYPES[doc] = "Formats of root filesystem images that you want to have created." +IMAGE_FSTYPES_DEBUGFS[doc] = "Formats of the debug root filesystem images that you want to have created." +IMAGE_GEN_DEBUGFS[doc] = "When set to '1', generate a companion debug object/source filesystem image." IMAGE_INSTALL[doc] = "Specifies the packages to install into an image. Image recipes set IMAGE_INSTALL to specify the packages to install into an image through image.bbclass." IMAGE_LINGUAS[doc] = "Specifies the list of locales to install into the image during the root filesystem construction process." IMAGE_NAME[doc] = "The name of the output image files minus the extension." diff --git a/meta/conf/local.conf.sample.extended b/meta/conf/local.conf.sample.extended index ccdd326..bc765a1 100644 --- a/meta/conf/local.conf.sample.extended +++ b/meta/conf/local.conf.sample.extended @@ -165,6 +165,15 @@ # currently an example class is image_types_uboot # IMAGE_CLASSES = " image_types_uboot" +# The following options will build a companion 'debug filesystem' in addition +# to the normal deployable filesystem. This companion system allows a +# debugger to know the symbols and related sources. It can be used to +# debug a remote 'production' system without having to add the debug symbols +# and sources to remote system. If IMAGE_FSTYPES_DEBUGFS is not defined, it +# defaults to IMAGE_FSTYPES. +#IMAGE_GEN_DEBUGFS = "1" +#IMAGE_FSTYPES_DEBUGFS = "tar.gz" + # Incremental rpm image generation, the rootfs would be totally removed # and re-created in the second generation by default, but with # INC_RPM_IMAGE_GEN = "1", the rpm based rootfs would be kept, and will diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py index f9e9bfd..688df9d 100644 --- a/meta/lib/oe/image.py +++ b/meta/lib/oe/image.py @@ -373,6 +373,43 @@ class Image(ImageDepGraph): bb.note("Creating symlinks for %s image ..." % image_type) self._create_symlinks(subimages) + # Process the debug filesystem... + if self.d.getVar('IMAGE_GEN_DEBUGFS', True) == "1": + bb.note("Processing debugfs image(s) ...") + orig_d = self.d + debugfs_d = bb.data.createCopy(self.d) + self.d = debugfs_d + + self.d.setVar('IMAGE_ROOTFS', orig_d.getVar('IMAGE_ROOTFS', True) + '-dbg') + self.d.setVar('IMAGE_NAME', orig_d.getVar('IMAGE_NAME', True) + '-dbg') + self.d.setVar('IMAGE_LINK_NAME', orig_d.getVar('IMAGE_LINK_NAME', True) + '-dbg') + + debugfs_image_fstypes = orig_d.getVar('IMAGE_FSTYPES_DEBUGFS', True) + if debugfs_image_fstypes: + self.d.setVar('IMAGE_FSTYPES', orig_d.getVar('IMAGE_FSTYPES_DEBUGFS', True)) + + self._remove_old_symlinks() + + debugfs_image_cmd_groups = self._get_imagecmds() + + for image_cmds in debugfs_image_cmd_groups: + # create the images in parallel + nproc = multiprocessing.cpu_count() + pool = bb.utils.multiprocessingpool(nproc) + results = list(pool.imap(generate_image, image_cmds)) + pool.close() + pool.join() + + for result in results: + if result is not None: + bb.fatal(result) + + for image_type, subimages, script in image_cmds: + bb.note("Creating debugfs symlinks for %s image ..." % image_type) + self._create_symlinks(subimages) + + self.d = orig_d + execute_pre_post_process(self.d, post_process_cmds) -- 1.9.3 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation 2015-10-01 18:26 [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation Mark Hatle 2015-10-01 18:26 ` [PATCH] lib/oe/image.py: Add image generation for companion debug filesystem Mark Hatle @ 2015-10-01 22:31 ` Richard Purdie 2015-10-02 14:23 ` Mark Hatle 1 sibling, 1 reply; 6+ messages in thread From: Richard Purdie @ 2015-10-01 22:31 UTC (permalink / raw) To: Mark Hatle; +Cc: david.reyna, openembedded-core On Thu, 2015-10-01 at 13:26 -0500, Mark Hatle wrote: > It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not > complete. The version that was merged back in May only contained the > filesystem generation pieces, but not the pieces for creating the image > from that filesystem. > > The code has been tested and is working. The only thing that I don't > particularly like is that the processing code and loop is a duplicate of > the code that runs just before. Unfortunately the only way around this > is to change the way that way the parallel bits are processed to support > multiple datastores.. (or create "another" function..) > > Any feedback appreciated, but without this the feature is broken! Could we not make a function which these two code points then call? Cheers, Richard ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation 2015-10-01 22:31 ` [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation Richard Purdie @ 2015-10-02 14:23 ` Mark Hatle 2015-10-02 14:30 ` Richard Purdie 0 siblings, 1 reply; 6+ messages in thread From: Mark Hatle @ 2015-10-02 14:23 UTC (permalink / raw) To: Richard Purdie; +Cc: david.reyna, openembedded-core On 10/1/15 5:31 PM, Richard Purdie wrote: > On Thu, 2015-10-01 at 13:26 -0500, Mark Hatle wrote: >> It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not >> complete. The version that was merged back in May only contained the >> filesystem generation pieces, but not the pieces for creating the image >> from that filesystem. >> >> The code has been tested and is working. The only thing that I don't >> particularly like is that the processing code and loop is a duplicate of >> the code that runs just before. Unfortunately the only way around this >> is to change the way that way the parallel bits are processed to support >> multiple datastores.. (or create "another" function..) >> >> Any feedback appreciated, but without this the feature is broken! > > Could we not make a function which these two code points then call? The duplicate piece is because the existing setup and loop depend on the local self.d value(s). In order to do this, we need to temporarily modify self.d and run this under an alternative datastore, and then put it back to the original value. If I don't duplicate the: for image_cmds in debugfs_image_cmd_groups: ... results = list(pool.imap(generate_image, image_cmds)) ... for image_type, subimages, script in image_cmds: bb.note("Creating debugfs symlinks for %s image ..." % image_type) self._create_symlinks(subimages) there is no concept of two different datastores. The alternative we have is to include a reference to the datastore itself in the 'image_cmds'. Then we could support any number of datastores as appropriate for the commands. (This of course will require additional changes to be able to pass that datastore to the various users.) --Mark > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation 2015-10-02 14:23 ` Mark Hatle @ 2015-10-02 14:30 ` Richard Purdie 2015-10-02 14:50 ` Mark Hatle 0 siblings, 1 reply; 6+ messages in thread From: Richard Purdie @ 2015-10-02 14:30 UTC (permalink / raw) To: Mark Hatle; +Cc: david.reyna, openembedded-core On Fri, 2015-10-02 at 09:23 -0500, Mark Hatle wrote: > On 10/1/15 5:31 PM, Richard Purdie wrote: > > On Thu, 2015-10-01 at 13:26 -0500, Mark Hatle wrote: > >> It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not > >> complete. The version that was merged back in May only contained the > >> filesystem generation pieces, but not the pieces for creating the image > >> from that filesystem. > >> > >> The code has been tested and is working. The only thing that I don't > >> particularly like is that the processing code and loop is a duplicate of > >> the code that runs just before. Unfortunately the only way around this > >> is to change the way that way the parallel bits are processed to support > >> multiple datastores.. (or create "another" function..) > >> > >> Any feedback appreciated, but without this the feature is broken! > > > > Could we not make a function which these two code points then call? > > The duplicate piece is because the existing setup and loop depend on the local > self.d value(s). In order to do this, we need to temporarily modify self.d and > run this under an alternative datastore, and then put it back to the original value. > > If I don't duplicate the: > > for image_cmds in debugfs_image_cmd_groups: > ... > results = list(pool.imap(generate_image, image_cmds)) > ... > for image_type, subimages, script in image_cmds: > bb.note("Creating debugfs symlinks for %s image ..." % > image_type) > self._create_symlinks(subimages) > > there is no concept of two different datastores. > > The alternative we have is to include a reference to the datastore itself in the > 'image_cmds'. Then we could support any number of datastores as appropriate for > the commands. (This of course will require additional changes to be able to > pass that datastore to the various users.) Ah, right. Going from memory, we can't share a datastore with a multiprocessing pool. We could do that with a multithreaded pool but I'm nervous about that for two reasons, one that we've had bad interaction between threads and processes in the past and secondly around locking and data usage in the threads. So not an easy problem... Cheers, Richard ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation 2015-10-02 14:30 ` Richard Purdie @ 2015-10-02 14:50 ` Mark Hatle 0 siblings, 0 replies; 6+ messages in thread From: Mark Hatle @ 2015-10-02 14:50 UTC (permalink / raw) To: Richard Purdie; +Cc: david.reyna, openembedded-core On 10/2/15 9:30 AM, Richard Purdie wrote: > On Fri, 2015-10-02 at 09:23 -0500, Mark Hatle wrote: >> On 10/1/15 5:31 PM, Richard Purdie wrote: >>> On Thu, 2015-10-01 at 13:26 -0500, Mark Hatle wrote: >>>> It was noticed today that the IMAGE_GEN_DEBUGFS implementation was not >>>> complete. The version that was merged back in May only contained the >>>> filesystem generation pieces, but not the pieces for creating the image >>>> from that filesystem. >>>> >>>> The code has been tested and is working. The only thing that I don't >>>> particularly like is that the processing code and loop is a duplicate of >>>> the code that runs just before. Unfortunately the only way around this >>>> is to change the way that way the parallel bits are processed to support >>>> multiple datastores.. (or create "another" function..) >>>> >>>> Any feedback appreciated, but without this the feature is broken! >>> >>> Could we not make a function which these two code points then call? >> >> The duplicate piece is because the existing setup and loop depend on the local >> self.d value(s). In order to do this, we need to temporarily modify self.d and >> run this under an alternative datastore, and then put it back to the original value. >> >> If I don't duplicate the: >> >> for image_cmds in debugfs_image_cmd_groups: >> ... >> results = list(pool.imap(generate_image, image_cmds)) >> ... >> for image_type, subimages, script in image_cmds: >> bb.note("Creating debugfs symlinks for %s image ..." % >> image_type) >> self._create_symlinks(subimages) >> >> there is no concept of two different datastores. >> >> The alternative we have is to include a reference to the datastore itself in the >> 'image_cmds'. Then we could support any number of datastores as appropriate for >> the commands. (This of course will require additional changes to be able to >> pass that datastore to the various users.) > > Ah, right. Going from memory, we can't share a datastore with a > multiprocessing pool. We could do that with a multithreaded pool but I'm > nervous about that for two reasons, one that we've had bad interaction > between threads and processes in the past and secondly around locking > and data usage in the threads. > > So not an easy problem... V2 coming.. it turns out you can avoid the multithread pool issue by making a few other select changes.. I missed this yesterday when I was floundering trying to get this to work. --Mark > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-10-02 14:50 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-10-01 18:26 [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation Mark Hatle 2015-10-01 18:26 ` [PATCH] lib/oe/image.py: Add image generation for companion debug filesystem Mark Hatle 2015-10-01 22:31 ` [PATCH] "Finish" the IMAGE_GEN_DEBUGFS implementation Richard Purdie 2015-10-02 14:23 ` Mark Hatle 2015-10-02 14:30 ` Richard Purdie 2015-10-02 14:50 ` Mark Hatle
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox