* [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)
@ 2014-02-03 11:38 Laurentiu Palcu
2014-02-03 11:54 ` Laurentiu Palcu
` (4 more replies)
0 siblings, 5 replies; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-03 11:38 UTC (permalink / raw)
To: openembedded-core
Hi all,
First off: this large patchset adds no change in the rootfs/image creation
functionality. The main logic is supposed to be the same as before. Also, I
decided not to spam the list with this large patchset (49 patches) and I sent
only the cover-letter. Those who want to take a peak at the code, can have a
look by clicking the link below.
If people would like the patches sent to the list too, just say so.
Ok, what's it all about then? Here are the main changes:
* all the old bash shell code (more then 90% of it anyway) has been ported to
Python. This would, hopefully, make the code more readable and allows us to
use some of the Python powerful features. The following new files were born,
together with a fair amount of changes in some bbclass'es:
> meta/lib/oe/manifest.py - implements the initial manifest creation (for now).
This is a file containing the list of packages going into the final image.
In theory, we should be able to use this file later to re-generate the
rootfs from a package feed, outside bitbake environment;
> meta/lib/oe/package_manager.py - implements the basic package management
API: install/remove packages, update index DB/index file creation. This
is mainly used by Rootfs/Sdk classes in order to generate the rootfs;
> meta/lib/oe/rootfs.py - contains code for rootfs generation;
> meta/lib/oe/image.py - the image creation part;
> meta/lib/oe/sdk.py - populate the SDK target/host rootfs.
With the new code, the entire do_rootfs routine in image.bbclass has now just
3 function calls: create_manifest(), create_rootfs(), create_image(). In this
order. Easier to debug just a certain part of the rootfs/image creation process.
* Some performance improvement has been obtained in 2 areas: index file creation
and image generation. These operations were serial in the old code. Now, they're
done in parallel.
* People complained about intercept scripts being hard to debug. That was because
the old code used a log checking routine that scanned the logs for some
standart error strings which were not necessarily errors. Hence, the intercepts
had their stderr redirected to /dev/null. This was now removed since we can
easily handle intercepts errors in python. So, errors will go to the log file.
That's about it. There is room for even further improvement, in the index creation
part. I had some code in place for that but I had to remove it, temporarily, due
to some synchronization issues. Hopefully, I'll find the issue and include that
piece back.
The code was tested on my machine and on a local AB for x86, x86-64, arm, ppc,
mips and build appliance. The builds and sanity tests finished successfully.
laurentiu
The following changes since commit b37dd451a52622d5b570183a81583cc34c2ff555:
rootfs_ipk: Ensure that BAD_RECOMMENDATIONS are honoured for all architectures (2014-02-02 22:37:42 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib lpalcu/rootfs_refactoring_ship_oecore
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/rootfs_refactoring_ship_oecore
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages
invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk
complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with
save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and
DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (31):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package
from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 109 +-
meta/classes/rootfs_rpm.bbclass | 184 ----
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1326 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 680 ++++++++++++
meta/lib/oe/sdk.py | 343 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 2965 insertions(+), 1949 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
--
1.7.9.5
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)
2014-02-03 11:38 [PATCH 00/49] Rootfs/image generation refactoring (cover letter only) Laurentiu Palcu
@ 2014-02-03 11:54 ` Laurentiu Palcu
2014-02-03 14:32 ` David Nyström
` (3 subsequent siblings)
4 siblings, 0 replies; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-03 11:54 UTC (permalink / raw)
To: openembedded-core
In my ignorance, I forgot to thank Hongxu, from WindRiver, for his help
on implementing the rpm related part.
Thanks Hongxu,
laurentiu
On Mon, Feb 03, 2014 at 01:38:35PM +0200, Laurentiu Palcu wrote:
> Hi all,
>
> First off: this large patchset adds no change in the rootfs/image creation
> functionality. The main logic is supposed to be the same as before. Also, I
> decided not to spam the list with this large patchset (49 patches) and I sent
> only the cover-letter. Those who want to take a peak at the code, can have a
> look by clicking the link below.
>
> If people would like the patches sent to the list too, just say so.
>
> Ok, what's it all about then? Here are the main changes:
> * all the old bash shell code (more then 90% of it anyway) has been ported to
> Python. This would, hopefully, make the code more readable and allows us to
> use some of the Python powerful features. The following new files were born,
> together with a fair amount of changes in some bbclass'es:
>
> > meta/lib/oe/manifest.py - implements the initial manifest creation (for now).
> This is a file containing the list of packages going into the final image.
> In theory, we should be able to use this file later to re-generate the
> rootfs from a package feed, outside bitbake environment;
>
> > meta/lib/oe/package_manager.py - implements the basic package management
> API: install/remove packages, update index DB/index file creation. This
> is mainly used by Rootfs/Sdk classes in order to generate the rootfs;
>
> > meta/lib/oe/rootfs.py - contains code for rootfs generation;
>
> > meta/lib/oe/image.py - the image creation part;
>
> > meta/lib/oe/sdk.py - populate the SDK target/host rootfs.
>
> With the new code, the entire do_rootfs routine in image.bbclass has now just
> 3 function calls: create_manifest(), create_rootfs(), create_image(). In this
> order. Easier to debug just a certain part of the rootfs/image creation process.
>
> * Some performance improvement has been obtained in 2 areas: index file creation
> and image generation. These operations were serial in the old code. Now, they're
> done in parallel.
>
> * People complained about intercept scripts being hard to debug. That was because
> the old code used a log checking routine that scanned the logs for some
> standart error strings which were not necessarily errors. Hence, the intercepts
> had their stderr redirected to /dev/null. This was now removed since we can
> easily handle intercepts errors in python. So, errors will go to the log file.
>
> That's about it. There is room for even further improvement, in the index creation
> part. I had some code in place for that but I had to remove it, temporarily, due
> to some synchronization issues. Hopefully, I'll find the issue and include that
> piece back.
>
> The code was tested on my machine and on a local AB for x86, x86-64, arm, ppc,
> mips and build appliance. The builds and sanity tests finished successfully.
>
> laurentiu
>
> The following changes since commit b37dd451a52622d5b570183a81583cc34c2ff555:
>
> rootfs_ipk: Ensure that BAD_RECOMMENDATIONS are honoured for all architectures (2014-02-02 22:37:42 +0000)
>
> are available in the git repository at:
>
> git://git.yoctoproject.org/poky-contrib lpalcu/rootfs_refactoring_ship_oecore
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/rootfs_refactoring_ship_oecore
>
> Hongxu Jia (18):
> lib/oe/manifest.py: add rpm image manifest creation
> lib/oe/package_manager.py: support RpmPM
> lib/oe/rootfs.py: support RpmRootfs
> lib/oe/rootfs.py: Tweak the rootfs removing operation
> rootfs.py: fix uninstall uneeded pkgs failed
> rootfs.py: strip kernel-abiversion useless readline
> lib/oe/image.py: fix get rootfs_extra_space failed
> image.bbclass/license.bbclass: ajustment list_installed_packages
> invoking
> license.bbclass: fix license_create_manifest failed
> lib/oe/sdk.py: support RpmRootfs
> image.bbclass/populate_sdk_base.bbclass: shift position of sdk
> complementary install definition
> Cleanup rootfs_rpm,package_rpm bbclass files
> lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
> lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
> lib/oe/rootfs.py: fix invoking _run_intercepts failed
> populate_sdk_rpm.bbclass: remove old shell code
> package_manger.py/rootfs.py: rename save_rpmpostinist with
> save_rpmpostinst
> lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and
> DB_CONFIG_CONTENT to RpmPM
>
> Laurentiu Palcu (31):
> lib/oe/utils.py: add pre/post process helper function
> lib/oe/image.py: add new image creation library
> lib/oe/manifest.py: add library for image manifest creation
> lib/oe/package_manager.py: add new package management library
> lib/oe/rootfs.py: add new library for rootfs creation
> Activate the new python rootfs/image creation routines
> oe/manifest.py: Add manifest creation for opkg
> package_(deb|ipk).bbclass: remove the stamp when creating package
> from cache
> lib/oe/manifest.py: create global variables for package types
> lib/oe/package_manager.py: add deploy dir locking mechanism
> lib/oe/package_manager.py: use the bitbake APT_ARGS variable
> lib/oe/package_manager.py: add support for opkg backend
> lib/oe/rootfs.py: add support for opkg backend
> Cleanup image,rootfs_ipk,package_ipk bbclass files
> image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
> lib/oe/manifest.py: create manifests for SDK too
> lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
> lib/oe/package_manager.py: fixes for dpkg backend
> lib/oe/sdk.py: add SDK class
> populate_sdk_base.bbclass: activate the new python routines
> image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
> populate_sdk_(deb|ipk).bbclass: remove old shell code
> lib/oe/package_manager.py: RpmPM fixes
> lib/oe/rootfs.py: warn the user if intercept hooks fail
> scripts/postinst-intercepts: do not redirect errors to /dev/null
> lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
> lib/oe/image.py: fix image size calculation routine
> lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
> lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
> lib/oe/package_manager.py: do not use stamps
> lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
>
> meta/classes/image.bbclass | 329 +-----
> meta/classes/image_types.bbclass | 113 --
> meta/classes/license.bbclass | 16 +-
> meta/classes/package_deb.bbclass | 151 +--
> meta/classes/package_ipk.bbclass | 202 +---
> meta/classes/package_rpm.bbclass | 472 --------
> meta/classes/populate_sdk_base.bbclass | 63 +-
> meta/classes/populate_sdk_deb.bbclass | 59 -
> meta/classes/populate_sdk_ipk.bbclass | 54 -
> meta/classes/populate_sdk_rpm.bbclass | 152 ---
> meta/classes/rootfs_deb.bbclass | 115 --
> meta/classes/rootfs_ipk.bbclass | 109 +-
> meta/classes/rootfs_rpm.bbclass | 184 ----
> meta/lib/oe/image.py | 239 ++++
> meta/lib/oe/manifest.py | 291 +++++
> meta/lib/oe/package_manager.py | 1326 +++++++++++++++++++++++
> meta/lib/oe/rootfs.py | 680 ++++++++++++
> meta/lib/oe/sdk.py | 343 ++++++
> meta/lib/oe/utils.py | 12 +-
> scripts/postinst-intercepts/update_font_cache | 2 +-
> scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
> 21 files changed, 2965 insertions(+), 1949 deletions(-)
> create mode 100644 meta/lib/oe/image.py
> create mode 100644 meta/lib/oe/manifest.py
> create mode 100644 meta/lib/oe/package_manager.py
> create mode 100644 meta/lib/oe/rootfs.py
> create mode 100644 meta/lib/oe/sdk.py
>
> --
> 1.7.9.5
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)
2014-02-03 11:38 [PATCH 00/49] Rootfs/image generation refactoring (cover letter only) Laurentiu Palcu
2014-02-03 11:54 ` Laurentiu Palcu
@ 2014-02-03 14:32 ` David Nyström
2014-02-03 16:47 ` Laurentiu Palcu
2014-02-05 10:07 ` [PATCH v2 00/50] " Laurentiu Palcu
` (2 subsequent siblings)
4 siblings, 1 reply; 22+ messages in thread
From: David Nyström @ 2014-02-03 14:32 UTC (permalink / raw)
To: Laurentiu Palcu, openembedded-core
On 2014-02-03 12:38, Laurentiu Palcu wrote:
> Hi all,
>
Have you tested that BUILD_IMAGES_FROM_FEEDS functionality is not broken
by these commits ?
Br,
David
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)
2014-02-03 14:32 ` David Nyström
@ 2014-02-03 16:47 ` Laurentiu Palcu
2014-02-04 14:08 ` David Nyström
0 siblings, 1 reply; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-03 16:47 UTC (permalink / raw)
To: David Nyström; +Cc: openembedded-core
On Mon, Feb 03, 2014 at 03:32:49PM +0100, David Nyström wrote:
> On 2014-02-03 12:38, Laurentiu Palcu wrote:
> >Hi all,
> >
>
> Have you tested that BUILD_IMAGES_FROM_FEEDS functionality is not
> broken by these commits ?
No, I did not test this. I tried to test it on master though and it
doesn't seem to work... Anyway, if you have a working environment, can
you give it a test? Technically, the feeds are added to the main
opkg.conf file in OPKG_PREPROCESS_CMD. And this functionality is
supposed to work.
laurentiu
>
> Br,
> David
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)
2014-02-03 16:47 ` Laurentiu Palcu
@ 2014-02-04 14:08 ` David Nyström
2014-02-04 14:49 ` David Nyström
0 siblings, 1 reply; 22+ messages in thread
From: David Nyström @ 2014-02-04 14:08 UTC (permalink / raw)
To: Laurentiu Palcu, David Nyström; +Cc: openembedded-core
On 2014-02-03 17:47, Laurentiu Palcu wrote:
> On Mon, Feb 03, 2014 at 03:32:49PM +0100, David Nyström wrote:
>> On 2014-02-03 12:38, Laurentiu Palcu wrote:
>>> Hi all,
>>>
>>
>> Have you tested that BUILD_IMAGES_FROM_FEEDS functionality is not
>> broken by these commits ?
> No, I did not test this. I tried to test it on master though and it
> doesn't seem to work...
Whats in master worked for me not too long ago with IPK at least.
> Anyway, if you have a working environment, can
> you give it a test? Technically, the feeds are added to the main
> opkg.conf file in OPKG_PREPROCESS_CMD. And this functionality is
> supposed to work.
>
> laurentiu
$ grep -r FEED conf/local.conf
BUILD_IMAGES_FROM_FEEDS="1"
IPK_FEED_URI="file:///media/sdb5/repo/build/tmp/deploy/"
$ bitbake -c rootfs core-image-minimal
DEBUG: Executing python function rootfs_process_ignore
DEBUG: Python function rootfs_process_ignore finished
DEBUG: Executing python function rootfs_runtime_mapping
DEBUG: Python function rootfs_runtime_mapping finished
DEBUG: Executing python function do_rootfs
NOTE: ###### Generate rootfs #######
ERROR: There are no packages in /media/sdb5/bld/tmp/deploy/ipk!
DEBUG: Python function do_rootfs finished
ERROR: Function failed: do_rootfs
/media/sdb5/bld/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.25006
(END)
$ grep -r "There are no packages in" ../poky-contrib/*
Binary file ../poky-contrib/meta/lib/oe/package_manager.pyc matches
../poky-contrib/meta/lib/oe/package_manager.py:
bb.fatal("There are no packages in %s" % self.deploy_dir)
../poky-contrib/meta/lib/oe/package_manager.py:
bb.fatal("There are no packages in %s!" % self.deploy_dir)
../poky-contrib/meta/lib/oe/package_manager.py:
bb.fatal("There are no packages in %s" % self.deploy_dir)
d
Br,
David
>
>>
>> Br,
>> David
>>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)
2014-02-04 14:08 ` David Nyström
@ 2014-02-04 14:49 ` David Nyström
2014-02-04 15:34 ` Laurentiu Palcu
0 siblings, 1 reply; 22+ messages in thread
From: David Nyström @ 2014-02-04 14:49 UTC (permalink / raw)
To: David Nyström, Laurentiu Palcu; +Cc: openembedded-core
On 2014-02-04 15:08, David Nyström wrote:
> On 2014-02-03 17:47, Laurentiu Palcu wrote:
>> On Mon, Feb 03, 2014 at 03:32:49PM +0100, David Nyström wrote:
>>> On 2014-02-03 12:38, Laurentiu Palcu wrote:
>>>> Hi all,
>>>>
>>>
>>> Have you tested that BUILD_IMAGES_FROM_FEEDS functionality is not
>>> broken by these commits ?
>> No, I did not test this. I tried to test it on master though and it
>> doesn't seem to work...
>
> Whats in master worked for me not too long ago with IPK at least.
>
>> Anyway, if you have a working environment, can
>> you give it a test? Technically, the feeds are added to the main
>> opkg.conf file in OPKG_PREPROCESS_CMD. And this functionality is
>> supposed to work.
>>
>> laurentiu
>
> $ grep -r FEED conf/local.conf
> BUILD_IMAGES_FROM_FEEDS="1"
> IPK_FEED_URI="file:///media/sdb5/repo/build/tmp/deploy/"
>
> $ bitbake -c rootfs core-image-minimal
>
> DEBUG: Executing python function rootfs_process_ignore
> DEBUG: Python function rootfs_process_ignore finished
> DEBUG: Executing python function rootfs_runtime_mapping
> DEBUG: Python function rootfs_runtime_mapping finished
> DEBUG: Executing python function do_rootfs
> NOTE: ###### Generate rootfs #######
> ERROR: There are no packages in /media/sdb5/bld/tmp/deploy/ipk!
> DEBUG: Python function do_rootfs finished
> ERROR: Function failed: do_rootfs
> /media/sdb5/bld/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.25006
> (END)
>
> $ grep -r "There are no packages in" ../poky-contrib/*
> Binary file ../poky-contrib/meta/lib/oe/package_manager.pyc matches
> ../poky-contrib/meta/lib/oe/package_manager.py: bb.fatal("There are no
> packages in %s" % self.deploy_dir)
> ../poky-contrib/meta/lib/oe/package_manager.py: bb.fatal("There are no
> packages in %s!" % self.deploy_dir)
> ../poky-contrib/meta/lib/oe/package_manager.py: bb.fatal("There are no
> packages in %s" % self.deploy_dir)
> d
>
> Br,
> David
>
Oops, bug on my behalf in above example, but it still fails the same way
when using a correct IPK_FEED_URI.
IPK_FEED_URIS += " \
all##file:///media/sdb5/repo/build/tmp/deploy/ipk/all \
${MACHINE}##file:///media/sdb5/repo/build/tmp/deploy/ipk/${MACHINE} \
${TARGET_ARCH}##file:///media/sdb5/repo/build/tmp/deploy/ipk/${TARGET_ARCH}
\
"
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)
2014-02-04 14:49 ` David Nyström
@ 2014-02-04 15:34 ` Laurentiu Palcu
2014-02-04 15:40 ` David Nyström
0 siblings, 1 reply; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-04 15:34 UTC (permalink / raw)
To: David Nyström; +Cc: openembedded-core
On Tue, Feb 04, 2014 at 03:49:25PM +0100, David Nyström wrote:
>
> Oops, bug on my behalf in above example, but it still fails the same
> way when using a correct IPK_FEED_URI.
>
> IPK_FEED_URIS += " \
> all##file:///media/sdb5/repo/build/tmp/deploy/ipk/all \
> ${MACHINE}##file:///media/sdb5/repo/build/tmp/deploy/ipk/${MACHINE} \
>
> ${TARGET_ARCH}##file:///media/sdb5/repo/build/tmp/deploy/ipk/${TARGET_ARCH}
> \
> "
Ok, I succeeded to check this in master. Apparently, I understood this
wrong. In my tests I was deleting the tmp/ and sstate-cache/ and I was
not expecting the target packages to be re-built. It turns out that this
entire process takes place but when the rootfs is created, the provided
feeds are used... I learned something new today! :)
Regarding the refactored code, indeed, it does not work in this case. It
always tries to create the index files in the deploy/ipk directory
before continuing with the rootfs creation. I guess a condition should
be added in the new code to check for BUILD_IMAGES_FROM_FEEDS and skip
the index creation step... Shouldn't be hard to fix.
laurentiu
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)
2014-02-04 15:34 ` Laurentiu Palcu
@ 2014-02-04 15:40 ` David Nyström
0 siblings, 0 replies; 22+ messages in thread
From: David Nyström @ 2014-02-04 15:40 UTC (permalink / raw)
To: Laurentiu Palcu, David Nyström; +Cc: openembedded-core
On 2014-02-04 16:34, Laurentiu Palcu wrote:
> On Tue, Feb 04, 2014 at 03:49:25PM +0100, David Nyström wrote:
>>
>> Oops, bug on my behalf in above example, but it still fails the same
>> way when using a correct IPK_FEED_URI.
>>
>> IPK_FEED_URIS += " \
>> all##file:///media/sdb5/repo/build/tmp/deploy/ipk/all \
>> ${MACHINE}##file:///media/sdb5/repo/build/tmp/deploy/ipk/${MACHINE} \
>>
>> ${TARGET_ARCH}##file:///media/sdb5/repo/build/tmp/deploy/ipk/${TARGET_ARCH}
>> \
>> "
> Ok, I succeeded to check this in master. Apparently, I understood this
> wrong. In my tests I was deleting the tmp/ and sstate-cache/ and I was
> not expecting the target packages to be re-built. It turns out that this
> entire process takes place but when the rootfs is created, the provided
> feeds are used... I learned something new today! :)
Yes, if you only run bitbake -c rootfs core-image-minimal, no target
packages _should_ be built.
It so happens that a few will still be built, for unknown reasons. But
this should be considered a bug.
native and cross packages will of course be built regardless.
> Regarding the refactored code, indeed, it does not work in this case. It
> always tries to create the index files in the deploy/ipk directory
> before continuing with the rootfs creation. I guess a condition should
> be added in the new code to check for BUILD_IMAGES_FROM_FEEDS and skip
> the index creation step... Shouldn't be hard to fix.
Thanks Laurentiu.
> laurentiu
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v2 00/50] Rootfs/image generation refactoring (cover letter only)
2014-02-03 11:38 [PATCH 00/49] Rootfs/image generation refactoring (cover letter only) Laurentiu Palcu
2014-02-03 11:54 ` Laurentiu Palcu
2014-02-03 14:32 ` David Nyström
@ 2014-02-05 10:07 ` Laurentiu Palcu
2014-02-05 10:58 ` David Nystrom
2014-02-05 15:05 ` [PATCH v3 00/51] " Laurentiu Palcu
2014-02-10 12:15 ` [PATCH v5 00/53] " Laurentiu Palcu
4 siblings, 1 reply; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-05 10:07 UTC (permalink / raw)
To: openembedded-core
V2:
* fix building from feeds for opkg;
* rebase;
Hi all,
First off: this large patchset adds no change in the rootfs/image creation
functionality. The main logic is supposed to be the same as before. Also, I
decided not to spam the list with this large patchset (49 patches) and I sent
only the cover-letter. Those who want to take a peak at the code, can have a
look by clicking the link below.
If people would like the patches sent to the list too, just say so.
Ok, what's it all about then? Here are the main changes:
* all the old bash shell code (more then 90% of it anyway) has been ported to
Python. This would, hopefully, make the code more readable and allows us to
use some of the Python powerful features. The following new files were born,
together with a fair amount of changes in some bbclass'es:
> meta/lib/oe/manifest.py - implements the initial manifest creation (for now).
This is a file containing the list of packages going into the final image.
In theory, we should be able to use this file later to re-generate the
rootfs from a package feed, outside bitbake environment;
> meta/lib/oe/package_manager.py - implements the basic package management
API: install/remove packages, update index DB/index file creation. This
is mainly used by Rootfs/Sdk classes in order to generate the rootfs;
> meta/lib/oe/rootfs.py - contains code for rootfs generation;
> meta/lib/oe/image.py - the image creation part;
> meta/lib/oe/sdk.py - populate the SDK target/host rootfs.
With the new code, the entire do_rootfs routine in image.bbclass has now just
3 function calls: create_manifest(), create_rootfs(), create_image(). In this
order. Easier to debug just a certain part of the rootfs/image creation process.
* Some performance improvement has been obtained in 2 areas: index file creation
and image generation. These operations were serial in the old code. Now, they're
done in parallel.
* People complained about intercept scripts being hard to debug. That was because
the old code used a log checking routine that scanned the logs for some
standart error strings which were not necessarily errors. Hence, the intercepts
had their stderr redirected to /dev/null. This was now removed since we can
easily handle intercepts errors in python. So, errors will go to the log file.
That's about it. There is room for even further improvement, in the index creation
part. I had some code in place for that but I had to remove it, temporarily, due
to some synchronization issues. Hopefully, I'll find the issue and include that
piece back.
The code was tested on my machine and on a local AB for x86, x86-64, arm, ppc,
mips and build appliance. The builds and sanity tests finished successfully.
laurentiu
The following changes since commit 4e9eb77108c7e628ebd00ec848b8bc034802e197:
sstate: Improve funciton checksums (2014-02-04 22:51:43 +0000)
are available in the git repository at:
git://mirror.rb.intel.com/git.yoctoproject.org/poky-contrib lpalcu/rootfs_refactoring_ship_oecore
for you to fetch changes up to a1b0b0bac55a18ca40b37bb91a84b27529db39ec:
rootfs.py, package_manager.py, sdk.py: Fix building from feeds feature for opkg (2014-02-05 11:51:57 +0200)
----------------------------------------------------------------
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (32):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
rootfs.py, package_manager.py, sdk.py: Fix building from feeds feature for opkg
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 138 +--
meta/classes/rootfs_rpm.bbclass | 184 ---
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1369 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 681 +++++++++++
meta/lib/oe/sdk.py | 345 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 3011 insertions(+), 1978 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages
invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk
complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with
save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and
DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (32):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package
from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
rootfs.py, package_manager.py, sdk.py: Fix building from feeds
feature for opkg
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 138 +--
meta/classes/rootfs_rpm.bbclass | 184 ---
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1369 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 681 +++++++++++
meta/lib/oe/sdk.py | 345 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 3011 insertions(+), 1978 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
--
1.7.9.5
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 00/50] Rootfs/image generation refactoring (cover letter only)
2014-02-05 10:07 ` [PATCH v2 00/50] " Laurentiu Palcu
@ 2014-02-05 10:58 ` David Nystrom
2014-02-05 12:09 ` David Nyström
0 siblings, 1 reply; 22+ messages in thread
From: David Nystrom @ 2014-02-05 10:58 UTC (permalink / raw)
To: Laurentiu Palcu; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 2716 bytes --]
On Wed, Feb 5, 2014 at 11:07 AM, Laurentiu Palcu
<laurentiu.palcu@intel.com>wrote:
> V2:
> * fix building from feeds for opkg;
> * rebase;
>
| NOTE: ###### Generate rootfs #######
| NOTE: Installing the following packages: run-postinsts
packagegroup-core-boot
| sed: -e expression #1, char 41: unterminated address regex
| sed: -e expression #1, char 42: unterminated address regex
| sed: -e expression #1, char 41: unterminated address regex
| sed: -e expression #1, char 42: unterminated address regex
[david] Why are these seds failing ?, I've seen this in master aswell
though.
| NOTE: Installing complementary packages ...
| NOTE: Installing the following packages:
| /media/sdb5/bld/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl: the
``install'' command requires at least one argument
| NOTE: Unable to install packages. Command
/media/sdb5/bld/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -f
/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/opkg.conf
-o
/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/rootfs
--force_postinstall --prefer-arch-to-version install returned 1
| NOTE: Running intercept scripts:
| NOTE: Executing write_package_manifest ...
| DEBUG: Executing python function write_package_manifest
| NOTE: Building from feeds activated!
| NOTE: Add all feed with URL
file:///media/sdb5/oel/build/tmp/deploy/ipk/all
| NOTE: Add qemuppc feed with URL
file:///media/sdb5/oel/build/tmp/deploy/ipk/qemuppc
| NOTE: Add ppc7400 feed with URL
file:///media/sdb5/oel/build/tmp/deploy/ipk/ppc7400
| DEBUG: Python function write_package_manifest finished
| NOTE: Executing license_create_manifest ...
| DEBUG: Executing shell function license_create_manifest
| ls: cannot access
/media/sdb5/bld/tmp/sysroots/qemuppc/pkgdata/runtime-reverse/base-files: No
such file or directory
| readlink: missing operand
| Try 'readlink --help' for more information.
| basename: missing operand
| Try 'basename --help' for more information.
| WARNING: exit code 1 from a shell command.
| DEBUG: Python function do_rootfs finished
| ERROR: Function failed: license_create_manifest (log file is located at
/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.29007)
ERROR: Task 6 (/media/sdb5/poky-contrib/meta/recipes-core/images/
core-image-minimal.bb, do_rootfs) failed with exit code '1'
NOTE: Tasks Summary: Attempted 504 tasks of which 503 didn't need to be
rerun and 1 failed.
Summary: 1 task failed:
/media/sdb5/poky-contrib/meta/recipes-core/images/core-image-minimal.bb,
do_rootfs
Summary: There was 1 ERROR message shown, returning a non-zero exit code
Br,
David
[-- Attachment #2: Type: text/html, Size: 3644 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 00/50] Rootfs/image generation refactoring (cover letter only)
2014-02-05 10:58 ` David Nystrom
@ 2014-02-05 12:09 ` David Nyström
2014-02-05 14:43 ` Laurentiu Palcu
0 siblings, 1 reply; 22+ messages in thread
From: David Nyström @ 2014-02-05 12:09 UTC (permalink / raw)
To: Laurentiu Palcu; +Cc: Patches and discussions about the oe-core layer
On 2014-02-05 11:58, David Nystrom wrote:
> On Wed, Feb 5, 2014 at 11:07 AM, Laurentiu Palcu
> <laurentiu.palcu@intel.com>wrote:
>
>> V2:
>> * fix building from feeds for opkg;
>> * rebase;
>>
>
> | NOTE: ###### Generate rootfs #######
> | NOTE: Installing the following packages: run-postinsts
> packagegroup-core-boot
> | sed: -e expression #1, char 41: unterminated address regex
> | sed: -e expression #1, char 42: unterminated address regex
> | sed: -e expression #1, char 41: unterminated address regex
> | sed: -e expression #1, char 42: unterminated address regex
>
> [david] Why are these seds failing ?, I've seen this in master aswell
> though.
After testing with the yocto 1.5 repo, the postinstall sed failures
where not seen anymore. Added Bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5789
>
> | NOTE: Installing complementary packages ...
> | NOTE: Installing the following packages:
> | /media/sdb5/bld/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl: the
> ``install'' command requires at least one argument
> | NOTE: Unable to install packages. Command
> /media/sdb5/bld/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -f
> /media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/opkg.conf
> -o
> /media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/rootfs
> --force_postinstall --prefer-arch-to-version install returned 1
> | NOTE: Running intercept scripts:
> | NOTE: Executing write_package_manifest ...
> | DEBUG: Executing python function write_package_manifest
> | NOTE: Building from feeds activated!
> | NOTE: Add all feed with URL
> file:///media/sdb5/oel/build/tmp/deploy/ipk/all
> | NOTE: Add qemuppc feed with URL
> file:///media/sdb5/oel/build/tmp/deploy/ipk/qemuppc
> | NOTE: Add ppc7400 feed with URL
> file:///media/sdb5/oel/build/tmp/deploy/ipk/ppc7400
> | DEBUG: Python function write_package_manifest finished
> | NOTE: Executing license_create_manifest ...
> | DEBUG: Executing shell function license_create_manifest
> | ls: cannot access
> /media/sdb5/bld/tmp/sysroots/qemuppc/pkgdata/runtime-reverse/base-files: No
> such file or directory
> | readlink: missing operand
> | Try 'readlink --help' for more information.
> | basename: missing operand
> | Try 'basename --help' for more information.
> | WARNING: exit code 1 from a shell command.
> | DEBUG: Python function do_rootfs finished
> | ERROR: Function failed: license_create_manifest (log file is located at
> /media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.29007)
> ERROR: Task 6 (/media/sdb5/poky-contrib/meta/recipes-core/images/
> core-image-minimal.bb, do_rootfs) failed with exit code '1'
> NOTE: Tasks Summary: Attempted 504 tasks of which 503 didn't need to be
> rerun and 1 failed.
>
> Summary: 1 task failed:
> /media/sdb5/poky-contrib/meta/recipes-core/images/core-image-minimal.bb,
> do_rootfs
> Summary: There was 1 ERROR message shown, returning a non-zero exit code
>
> Br,
> David
>
FYI: to easily recreate this testcase:
-Clean build env-
$ grep -C 6 -r FEEDS conf/local.conf
# Default to setting automatically based on cpu count
PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
#
# For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE = "-j
4" would
# be appropriate for example.
BUILD_IMAGES_FROM_FEEDS="1"
IPK_FEED_URIS = " \
all##http://downloads.yoctoproject.org/releases/yocto/yocto-1.5/ipk/all \
${MACHINE}##http://downloads.yoctoproject.org/releases/yocto/yocto-1.5/ipk/${MACHINE}
\
ppc7400##http://downloads.yoctoproject.org/releases/yocto/yocto-1.5/ipk/ppc7400
\
"
PACKAGE_CLASSES="ipk" MACHINE="qemuppc" bitbake -c rootfs core-image-minimal
Br,
David
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 00/50] Rootfs/image generation refactoring (cover letter only)
2014-02-05 12:09 ` David Nyström
@ 2014-02-05 14:43 ` Laurentiu Palcu
2014-02-05 18:00 ` David Nystrom
0 siblings, 1 reply; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-05 14:43 UTC (permalink / raw)
To: David Nyström; +Cc: Patches and discussions about the oe-core layer
On Wed, Feb 05, 2014 at 01:09:05PM +0100, David Nyström wrote:
> >
> >| NOTE: Installing complementary packages ...
> >| NOTE: Installing the following packages:
> >| /media/sdb5/bld/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl: the
> >``install'' command requires at least one argument
> >| NOTE: Unable to install packages. Command
> >/media/sdb5/bld/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -f
> >/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/opkg.conf
> >-o
> >/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/rootfs
> >--force_postinstall --prefer-arch-to-version install returned 1
I'll send a quick fix for this. This, however, is not the reason of the
build failure. It's complementary packages that are installed and an
empty list is passed to install(). Since these are attempt only
packages, the build does not fail because of this. Anyway, I added a
simple check and I will send a v3.
> >| NOTE: Running intercept scripts:
> >| NOTE: Executing write_package_manifest ...
> >| DEBUG: Executing python function write_package_manifest
> >| NOTE: Building from feeds activated!
> >| NOTE: Add all feed with URL
> >file:///media/sdb5/oel/build/tmp/deploy/ipk/all
> >| NOTE: Add qemuppc feed with URL
> >file:///media/sdb5/oel/build/tmp/deploy/ipk/qemuppc
> >| NOTE: Add ppc7400 feed with URL
> >file:///media/sdb5/oel/build/tmp/deploy/ipk/ppc7400
> >| DEBUG: Python function write_package_manifest finished
> >| NOTE: Executing license_create_manifest ...
> >| DEBUG: Executing shell function license_create_manifest
> >| ls: cannot access
> >/media/sdb5/bld/tmp/sysroots/qemuppc/pkgdata/runtime-reverse/base-files: No
> >such file or directory
> >| readlink: missing operand
> >| Try 'readlink --help' for more information.
> >| basename: missing operand
> >| Try 'basename --help' for more information.
> >| WARNING: exit code 1 from a shell command.
> >| DEBUG: Python function do_rootfs finished
> >| ERROR: Function failed: license_create_manifest (log file is located at
> >/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.29007)
Now, this is the interesting part which, btw, happens on master too. That's
because you're installing from feeds and you're installing an older
version that has different packages than the latest (these are read from
package.manifest). The YP 1.5 core-image-minimal package.manifest
contains update-alternatives-cworth. The latest core-image-minimal does
not contain this package anymore. It contains update-alternatives-opkg.
The easiest way would be to add a check in license_create_manifest() to
just ignore packages that cannot be found. However, since we're dealing
with licenses here, I honestly don't know if this is the right way of
doing it...
laurentiu
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v3 00/51] Rootfs/image generation refactoring (cover letter only)
2014-02-03 11:38 [PATCH 00/49] Rootfs/image generation refactoring (cover letter only) Laurentiu Palcu
` (2 preceding siblings ...)
2014-02-05 10:07 ` [PATCH v2 00/50] " Laurentiu Palcu
@ 2014-02-05 15:05 ` Laurentiu Palcu
2014-02-10 7:53 ` [PATCH v4 00/52] " Laurentiu Palcu
2014-02-10 12:15 ` [PATCH v5 00/53] " Laurentiu Palcu
4 siblings, 1 reply; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-05 15:05 UTC (permalink / raw)
To: openembedded-core
V3:
* add simple check to just return if we're attempting to install packages and
the list is empty;
V2:
* fix building from feeds for opkg;
* rebase;
Hi all,
First off: this large patchset adds no change in the rootfs/image creation
functionality. The main logic is supposed to be the same as before. Also, I
decided not to spam the list with this large patchset (49 patches) and I sent
only the cover-letter. Those who want to take a peak at the code, can have a
look by clicking the link below.
If people would like the patches sent to the list too, just say so.
Ok, what's it all about then? Here are the main changes:
* all the old bash shell code (more then 90% of it anyway) has been ported to
Python. This would, hopefully, make the code more readable and allows us to
use some of the Python powerful features. The following new files were born,
together with a fair amount of changes in some bbclass'es:
> meta/lib/oe/manifest.py - implements the initial manifest creation (for now).
This is a file containing the list of packages going into the final image.
In theory, we should be able to use this file later to re-generate the
rootfs from a package feed, outside bitbake environment;
> meta/lib/oe/package_manager.py - implements the basic package management
API: install/remove packages, update index DB/index file creation. This
is mainly used by Rootfs/Sdk classes in order to generate the rootfs;
> meta/lib/oe/rootfs.py - contains code for rootfs generation;
> meta/lib/oe/image.py - the image creation part;
> meta/lib/oe/sdk.py - populate the SDK target/host rootfs.
With the new code, the entire do_rootfs routine in image.bbclass has now just
3 function calls: create_manifest(), create_rootfs(), create_image(). In this
order. Easier to debug just a certain part of the rootfs/image creation process.
* Some performance improvement has been obtained in 2 areas: index file creation
and image generation. These operations were serial in the old code. Now, they're
done in parallel.
* People complained about intercept scripts being hard to debug. That was because
the old code used a log checking routine that scanned the logs for some
standart error strings which were not necessarily errors. Hence, the intercepts
had their stderr redirected to /dev/null. This was now removed since we can
easily handle intercepts errors in python. So, errors will go to the log file.
That's about it. There is room for even further improvement, in the index creation
part. I had some code in place for that but I had to remove it, temporarily, due
to some synchronization issues. Hopefully, I'll find the issue and include that
piece back.
The code was tested on my machine and on a local AB for x86, x86-64, arm, ppc,
mips and build appliance. The builds and sanity tests finished successfully.
laurentiu
The following changes since commit 4e9eb77108c7e628ebd00ec848b8bc034802e197:
sstate: Improve funciton checksums (2014-02-04 22:51:43 +0000)
are available in the git repository at:
git://mirror.rb.intel.com/git.yoctoproject.org/poky-contrib lpalcu/rootfs_refactoring_ship_oecore
for you to fetch changes up to 8a4a72dae5b3b9452f20d1c45fdf196ae450aaf7:
lib/oe/package_manager.py: check that package list is not empty (2014-02-05 16:57:52 +0200)
----------------------------------------------------------------
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (33):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
rootfs.py, package_manager.py, sdk.py: Fix building from feeds feature for opkg
lib/oe/package_manager.py: check that package list is not empty
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 138 +--
meta/classes/rootfs_rpm.bbclass | 184 ---
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1375 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 681 +++++++++++
meta/lib/oe/sdk.py | 345 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 3017 insertions(+), 1978 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages
invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk
complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with
save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and
DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (33):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package
from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
rootfs.py, package_manager.py, sdk.py: Fix building from feeds
feature for opkg
lib/oe/package_manager.py: check that package list is not empty
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 138 +--
meta/classes/rootfs_rpm.bbclass | 184 ---
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1375 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 681 +++++++++++
meta/lib/oe/sdk.py | 345 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 3017 insertions(+), 1978 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
--
1.7.9.5
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v2 00/50] Rootfs/image generation refactoring (cover letter only)
2014-02-05 14:43 ` Laurentiu Palcu
@ 2014-02-05 18:00 ` David Nystrom
0 siblings, 0 replies; 22+ messages in thread
From: David Nystrom @ 2014-02-05 18:00 UTC (permalink / raw)
To: Laurentiu Palcu; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 3705 bytes --]
On 5 Feb 2014 15:43, "Laurentiu Palcu" <laurentiu.palcu@intel.com> wrote:
>
> On Wed, Feb 05, 2014 at 01:09:05PM +0100, David Nyström wrote:
> > >
> > >| NOTE: Installing complementary packages ...
> > >| NOTE: Installing the following packages:
> > >| /media/sdb5/bld/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl: the
> > >``install'' command requires at least one argument
> > >| NOTE: Unable to install packages. Command
> > >/media/sdb5/bld/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -f
> >
>/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/opkg.conf
> > >-o
> >
>/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/rootfs
> > >--force_postinstall --prefer-arch-to-version install returned 1
> I'll send a quick fix for this. This, however, is not the reason of the
> build failure. It's complementary packages that are installed and an
> empty list is passed to install(). Since these are attempt only
> packages, the build does not fail because of this. Anyway, I added a
> simple check and I will send a v3.
>
> > >| NOTE: Running intercept scripts:
> > >| NOTE: Executing write_package_manifest ...
> > >| DEBUG: Executing python function write_package_manifest
> > >| NOTE: Building from feeds activated!
> > >| NOTE: Add all feed with URL
> > >file:///media/sdb5/oel/build/tmp/deploy/ipk/all
> > >| NOTE: Add qemuppc feed with URL
> > >file:///media/sdb5/oel/build/tmp/deploy/ipk/qemuppc
> > >| NOTE: Add ppc7400 feed with URL
> > >file:///media/sdb5/oel/build/tmp/deploy/ipk/ppc7400
> > >| DEBUG: Python function write_package_manifest finished
> > >| NOTE: Executing license_create_manifest ...
> > >| DEBUG: Executing shell function license_create_manifest
> > >| ls: cannot access
> >
>/media/sdb5/bld/tmp/sysroots/qemuppc/pkgdata/runtime-reverse/base-files: No
> > >such file or directory
> > >| readlink: missing operand
> > >| Try 'readlink --help' for more information.
> > >| basename: missing operand
> > >| Try 'basename --help' for more information.
> > >| WARNING: exit code 1 from a shell command.
> > >| DEBUG: Python function do_rootfs finished
> > >| ERROR: Function failed: license_create_manifest (log file is located
at
> >
>/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.29007)
> Now, this is the interesting part which, btw, happens on master too.
That's
> because you're installing from feeds and you're installing an older
> version that has different packages than the latest (these are read from
> package.manifest). The YP 1.5 core-image-minimal package.manifest
> contains update-alternatives-cworth. The latest core-image-minimal does
> not contain this package anymore. It contains update-alternatives-opkg.
Thanks Laurentiu,
The 1.5 repo reference was just to give you an easy testcase. Sorry for
creating confusion.
My debug run above was done with a repo created by poky on the master
branch.
>
> The easiest way would be to add a check in license_create_manifest() to
> just ignore packages that cannot be found. However, since we're dealing
> with licenses here, I honestly don't know if this is the right way of
> doing it...
Good point, I think it might be better to avoid creating a manifest, rather
than creating a potentially bogus manifest when using the FEEDS
functionality.
The per-package licenses should be already existing in the repo when using
a feed.
//DD
Br,
David
>
> laurentiu
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
[-- Attachment #2: Type: text/html, Size: 4707 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v4 00/52] Rootfs/image generation refactoring (cover letter only)
2014-02-05 15:05 ` [PATCH v3 00/51] " Laurentiu Palcu
@ 2014-02-10 7:53 ` Laurentiu Palcu
0 siblings, 0 replies; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-10 7:53 UTC (permalink / raw)
To: openembedded-core
V4:
* fix multilib build issue for RPM backend;
V3:
* add simple check to just return if we're attempting to install packages and
the list is empty;
V2:
* fix building from feeds for opkg;
* rebase;
Hi all,
First off: this large patchset adds no change in the rootfs/image creation
functionality. The main logic is supposed to be the same as before. Also, I
decided not to spam the list with this large patchset (49 patches) and I sent
only the cover-letter. Those who want to take a peak at the code, can have a
look by clicking the link below.
If people would like the patches sent to the list too, just say so.
Ok, what's it all about then? Here are the main changes:
* all the old bash shell code (more then 90% of it anyway) has been ported to
Python. This would, hopefully, make the code more readable and allows us to
use some of the Python powerful features. The following new files were born,
together with a fair amount of changes in some bbclass'es:
> meta/lib/oe/manifest.py - implements the initial manifest creation (for now).
This is a file containing the list of packages going into the final image.
In theory, we should be able to use this file later to re-generate the
rootfs from a package feed, outside bitbake environment;
> meta/lib/oe/package_manager.py - implements the basic package management
API: install/remove packages, update index DB/index file creation. This
is mainly used by Rootfs/Sdk classes in order to generate the rootfs;
> meta/lib/oe/rootfs.py - contains code for rootfs generation;
> meta/lib/oe/image.py - the image creation part;
> meta/lib/oe/sdk.py - populate the SDK target/host rootfs.
With the new code, the entire do_rootfs routine in image.bbclass has now just
3 function calls: create_manifest(), create_rootfs(), create_image(). In this
order. Easier to debug just a certain part of the rootfs/image creation process.
* Some performance improvement has been obtained in 2 areas: index file creation
and image generation. These operations were serial in the old code. Now, they're
done in parallel.
* People complained about intercept scripts being hard to debug. That was because
the old code used a log checking routine that scanned the logs for some
standart error strings which were not necessarily errors. Hence, the intercepts
had their stderr redirected to /dev/null. This was now removed since we can
easily handle intercepts errors in python. So, errors will go to the log file.
That's about it. There is room for even further improvement, in the index creation
part. I had some code in place for that but I had to remove it, temporarily, due
to some synchronization issues. Hopefully, I'll find the issue and include that
piece back.
The code was tested on my machine and on a local AB for x86, x86-64, arm, ppc,
mips and build appliance. The builds and sanity tests finished successfully.
laurentiu
The following changes since commit 4e9eb77108c7e628ebd00ec848b8bc034802e197:
sstate: Improve funciton checksums (2014-02-04 22:51:43 +0000)
are available in the git repository at:
git://mirror.rb.intel.com/git.yoctoproject.org/poky-contrib lpalcu/rootfs_refactoring_ship_oecore
for you to fetch changes up to b4891e7c5e02e687f037babc823cdbf7cff471a4:
lib/oe/package_manager.py: RpmPM: fix issue with multilib builds (2014-02-10 09:38:45 +0200)
----------------------------------------------------------------
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (34):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
rootfs.py, package_manager.py, sdk.py: Fix building from feeds feature for opkg
lib/oe/package_manager.py: check that package list is not empty
lib/oe/package_manager.py: RpmPM: fix issue with multilib builds
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 138 +--
meta/classes/rootfs_rpm.bbclass | 184 ---
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1374 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 681 +++++++++++
meta/lib/oe/sdk.py | 345 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 3016 insertions(+), 1978 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages
invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk
complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with
save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and
DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (34):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package
from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
rootfs.py, package_manager.py, sdk.py: Fix building from feeds
feature for opkg
lib/oe/package_manager.py: check that package list is not empty
lib/oe/package_manager.py: RpmPM: fix issue with multilib builds
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 138 +--
meta/classes/rootfs_rpm.bbclass | 184 ---
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1374 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 681 +++++++++++
meta/lib/oe/sdk.py | 345 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 3016 insertions(+), 1978 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
--
1.7.9.5
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v5 00/53] Rootfs/image generation refactoring (cover letter only)
2014-02-03 11:38 [PATCH 00/49] Rootfs/image generation refactoring (cover letter only) Laurentiu Palcu
` (3 preceding siblings ...)
2014-02-05 15:05 ` [PATCH v3 00/51] " Laurentiu Palcu
@ 2014-02-10 12:15 ` Laurentiu Palcu
2014-02-11 8:42 ` David Nyström
2014-02-11 14:33 ` David Nyström
4 siblings, 2 replies; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-10 12:15 UTC (permalink / raw)
To: openembedded-core
V5:
* dump command output in case errors occur;
V4:
* fix multilib build issue for RPM backend;
V3:
* add simple check to just return if we're attempting to install packages and
the list is empty;
V2:
* fix building from feeds for opkg;
* rebase;
Hi all,
First off: this large patchset adds no change in the rootfs/image creation
functionality. The main logic is supposed to be the same as before. Also, I
decided not to spam the list with this large patchset (49 patches) and I sent
only the cover-letter. Those who want to take a peak at the code, can have a
look by clicking the link below.
If people would like the patches sent to the list too, just say so.
Ok, what's it all about then? Here are the main changes:
* all the old bash shell code (more then 90% of it anyway) has been ported to
Python. This would, hopefully, make the code more readable and allows us to
use some of the Python powerful features. The following new files were born,
together with a fair amount of changes in some bbclass'es:
> meta/lib/oe/manifest.py - implements the initial manifest creation (for now).
This is a file containing the list of packages going into the final image.
In theory, we should be able to use this file later to re-generate the
rootfs from a package feed, outside bitbake environment;
> meta/lib/oe/package_manager.py - implements the basic package management
API: install/remove packages, update index DB/index file creation. This
is mainly used by Rootfs/Sdk classes in order to generate the rootfs;
> meta/lib/oe/rootfs.py - contains code for rootfs generation;
> meta/lib/oe/image.py - the image creation part;
> meta/lib/oe/sdk.py - populate the SDK target/host rootfs.
With the new code, the entire do_rootfs routine in image.bbclass has now just
3 function calls: create_manifest(), create_rootfs(), create_image(). In this
order. Easier to debug just a certain part of the rootfs/image creation process.
* Some performance improvement has been obtained in 2 areas: index file creation
and image generation. These operations were serial in the old code. Now, they're
done in parallel.
* People complained about intercept scripts being hard to debug. That was because
the old code used a log checking routine that scanned the logs for some
standart error strings which were not necessarily errors. Hence, the intercepts
had their stderr redirected to /dev/null. This was now removed since we can
easily handle intercepts errors in python. So, errors will go to the log file.
That's about it. There is room for even further improvement, in the index creation
part. I had some code in place for that but I had to remove it, temporarily, due
to some synchronization issues. Hopefully, I'll find the issue and include that
piece back.
The code was tested on my machine and on a local AB for x86, x86-64, arm, ppc,
mips and build appliance. The builds and sanity tests finished successfully.
laurentiu
The following changes since commit 4e9eb77108c7e628ebd00ec848b8bc034802e197:
sstate: Improve funciton checksums (2014-02-04 22:51:43 +0000)
are available in the git repository at:
git://mirror.rb.intel.com/git.yoctoproject.org/poky-contrib lpalcu/rootfs_refactoring_ship_oecore
for you to fetch changes up to 20104ecbb94a1ada0cb3a26c7801b56a897dc900:
image.py, package_manager.py, rootfs.py: dump command output on error (2014-02-10 14:06:08 +0200)
----------------------------------------------------------------
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (35):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
rootfs.py, package_manager.py, sdk.py: Fix building from feeds feature for opkg
lib/oe/package_manager.py: check that package list is not empty
lib/oe/package_manager.py: RpmPM: fix issue with multilib builds
image.py, package_manager.py, rootfs.py: dump command output on error
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 138 +--
meta/classes/rootfs_rpm.bbclass | 184 ---
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1375 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 681 +++++++++++
meta/lib/oe/sdk.py | 345 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 3017 insertions(+), 1978 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
Hongxu Jia (18):
lib/oe/manifest.py: add rpm image manifest creation
lib/oe/package_manager.py: support RpmPM
lib/oe/rootfs.py: support RpmRootfs
lib/oe/rootfs.py: Tweak the rootfs removing operation
rootfs.py: fix uninstall uneeded pkgs failed
rootfs.py: strip kernel-abiversion useless readline
lib/oe/image.py: fix get rootfs_extra_space failed
image.bbclass/license.bbclass: ajustment list_installed_packages
invoking
license.bbclass: fix license_create_manifest failed
lib/oe/sdk.py: support RpmRootfs
image.bbclass/populate_sdk_base.bbclass: shift position of sdk
complementary install definition
Cleanup rootfs_rpm,package_rpm bbclass files
lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
lib/oe/rootfs.py: fix invoking _run_intercepts failed
populate_sdk_rpm.bbclass: remove old shell code
package_manger.py/rootfs.py: rename save_rpmpostinist with
save_rpmpostinst
lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and
DB_CONFIG_CONTENT to RpmPM
Laurentiu Palcu (35):
lib/oe/utils.py: add pre/post process helper function
lib/oe/image.py: add new image creation library
lib/oe/manifest.py: add library for image manifest creation
lib/oe/package_manager.py: add new package management library
lib/oe/rootfs.py: add new library for rootfs creation
Activate the new python rootfs/image creation routines
oe/manifest.py: Add manifest creation for opkg
package_(deb|ipk).bbclass: remove the stamp when creating package
from cache
lib/oe/manifest.py: create global variables for package types
lib/oe/package_manager.py: add deploy dir locking mechanism
lib/oe/package_manager.py: use the bitbake APT_ARGS variable
lib/oe/package_manager.py: add support for opkg backend
lib/oe/rootfs.py: add support for opkg backend
Cleanup image,rootfs_ipk,package_ipk bbclass files
image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
lib/oe/manifest.py: create manifests for SDK too
lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
lib/oe/package_manager.py: fixes for dpkg backend
lib/oe/sdk.py: add SDK class
populate_sdk_base.bbclass: activate the new python routines
image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
populate_sdk_(deb|ipk).bbclass: remove old shell code
lib/oe/package_manager.py: RpmPM fixes
lib/oe/rootfs.py: warn the user if intercept hooks fail
scripts/postinst-intercepts: do not redirect errors to /dev/null
lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
lib/oe/image.py: fix image size calculation routine
lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
lib/oe/package_manager.py: do not use stamps
lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
rootfs.py, package_manager.py, sdk.py: Fix building from feeds
feature for opkg
lib/oe/package_manager.py: check that package list is not empty
lib/oe/package_manager.py: RpmPM: fix issue with multilib builds
image.py, package_manager.py, rootfs.py: dump command output on error
meta/classes/image.bbclass | 329 +-----
meta/classes/image_types.bbclass | 113 --
meta/classes/license.bbclass | 16 +-
meta/classes/package_deb.bbclass | 151 +--
meta/classes/package_ipk.bbclass | 202 +---
meta/classes/package_rpm.bbclass | 472 --------
meta/classes/populate_sdk_base.bbclass | 63 +-
meta/classes/populate_sdk_deb.bbclass | 59 -
meta/classes/populate_sdk_ipk.bbclass | 54 -
meta/classes/populate_sdk_rpm.bbclass | 152 ---
meta/classes/rootfs_deb.bbclass | 115 --
meta/classes/rootfs_ipk.bbclass | 138 +--
meta/classes/rootfs_rpm.bbclass | 184 ---
meta/lib/oe/image.py | 239 ++++
meta/lib/oe/manifest.py | 291 +++++
meta/lib/oe/package_manager.py | 1375 +++++++++++++++++++++++
meta/lib/oe/rootfs.py | 681 +++++++++++
meta/lib/oe/sdk.py | 345 ++++++
meta/lib/oe/utils.py | 12 +-
scripts/postinst-intercepts/update_font_cache | 2 +-
scripts/postinst-intercepts/update_pixbuf_cache | 2 +-
21 files changed, 3017 insertions(+), 1978 deletions(-)
create mode 100644 meta/lib/oe/image.py
create mode 100644 meta/lib/oe/manifest.py
create mode 100644 meta/lib/oe/package_manager.py
create mode 100644 meta/lib/oe/rootfs.py
create mode 100644 meta/lib/oe/sdk.py
--
1.7.9.5
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v5 00/53] Rootfs/image generation refactoring (cover letter only)
2014-02-10 12:15 ` [PATCH v5 00/53] " Laurentiu Palcu
@ 2014-02-11 8:42 ` David Nyström
2014-02-11 8:50 ` Laurentiu Palcu
2014-02-11 14:33 ` David Nyström
1 sibling, 1 reply; 22+ messages in thread
From: David Nyström @ 2014-02-11 8:42 UTC (permalink / raw)
To: Laurentiu Palcu, openembedded-core
On 2014-02-10 13:15, Laurentiu Palcu wrote:
> V5:
> * dump command output in case errors occur;
>
> V4:
> * fix multilib build issue for RPM backend;
>
> V3:
> * add simple check to just return if we're attempting to install packages and
> the list is empty;
>
> V2:
> * fix building from feeds for opkg;
> * rebase;
>
When stubbing license manifest creation for the FEEDS functionality:
diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index c668783..a0413b9 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -26,6 +26,12 @@ python write_package_manifest() {
}
license_create_manifest() {
+ # Test if BUILD_IMAGES_FROM_FEEDS is defined in env
+ echo ${BUILD_IMAGES_FROM_FEEDS}
+ if [ -n "${BUILD_IMAGES_FROM_FEEDS}" ]; then
+ exit 0
+ fi
+
I get this error message on v5.
Br,
David
$ bitbake -c rootfs core-image-minimal
Parsing recipes: 100%
|######################################################################################################################################################################|
Time: 00:00:12
Parsing of 852 .bb files complete (0 cached, 852 parsed). 1212 targets,
56 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.21.1"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "Ubuntu-13.10"
TARGET_SYS = "powerpc-poky-linux"
MACHINE = "qemuppc"
DISTRO = "poky"
DISTRO_VERSION = "1.5+snapshot-20140211"
TUNE_FEATURES = "m32 fpu-hard ppc7400 altivec"
TARGET_FPU = ""
meta
meta-yocto
meta-yocto-bsp =
"(detachedfromFETCH_HEAD):20104ecbb94a1ada0cb3a26c7801b56a897dc900"
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: Error executing a python function in
/media/sdb5/poky-contrib/meta/recipes-core/images/core-image-minimal.bb:
The stack trace of python calls that resulted in this exception/failure was:
File: 'do_rootfs', lineno: 17, function: <module>
0013: # generate final images
0014: create_image(d)
0015:
0016:
*** 0017:do_rootfs(d)
0018:
File: 'do_rootfs', lineno: 14, function: do_rootfs
0010: # generate rootfs
0011: create_rootfs(d)
0012:
0013: # generate final images
*** 0014: create_image(d)
0015:
0016:
0017:do_rootfs(d)
0018:
File: '/media/sdb5/poky-contrib/meta/lib/oe/image.py', lineno: 231,
function: create_image
0227: execute_pre_post_process(self.d, post_process_cmds)
0228:
0229:
0230:def create_image(d):
*** 0231: Image(d).create()
0232:
0233:if __name__ == "__main__":
0234: """
0235: Image creation can be called independent from bitbake
environment.
File: '/media/sdb5/poky-contrib/meta/lib/oe/image.py', lineno: 210,
function: create
0206: execute_pre_post_process(self.d, pre_process_cmds)
0207:
0208: self._remove_old_symlinks()
0209:
*** 0210: image_cmds = self._get_imagecmds()
0211:
0212: # create the images in parallel
0213: nproc = multiprocessing.cpu_count()
0214: pool = bb.utils.multiprocessingpool(nproc)
File: '/media/sdb5/poky-contrib/meta/lib/oe/image.py', lineno: 195,
function: _get_imagecmds
0191: cmds.append(localdata.expand("\trm
${IMAGE_NAME}.rootfs.${type}"))
0192: else:
0193: subimages.append(type)
0194:
*** 0195: script_name = self._write_script(type, cmds)
0196:
0197: image_cmds.append((type, subimages, script_name))
0198:
0199: return image_cmds
File: '/media/sdb5/poky-contrib/meta/lib/oe/image.py', lineno: 158,
function: _write_script
0154: self.d.setVarFlag('img_creation_func', 'func', 1)
0155: self.d.setVarFlag('img_creation_func', 'fakeroot', 1)
0156:
0157: with open(script_name, "w+") as script:
*** 0158: script.write("%s" % bb.build.shell_trap_code())
0159: script.write("export ROOTFS_SIZE=%d\n" %
self._get_rootfs_size())
0160: bb.data.emit_func('img_creation_func', script,
self.d)
0161: script.write("img_creation_func\n")
0162:
Exception: AttributeError: 'module' object has no attribute
'shell_trap_code'
ERROR: Function failed: do_rootfs
ERROR: Logfile of failure stored in:
/media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.9994
ERROR: Task 6
(/media/sdb5/poky-contrib/meta/recipes-core/images/core-image-minimal.bb, do_rootfs)
failed with exit code '1'
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH v5 00/53] Rootfs/image generation refactoring (cover letter only)
2014-02-11 8:42 ` David Nyström
@ 2014-02-11 8:50 ` Laurentiu Palcu
2014-02-11 8:56 ` David Nyström
0 siblings, 1 reply; 22+ messages in thread
From: Laurentiu Palcu @ 2014-02-11 8:50 UTC (permalink / raw)
To: David Nyström; +Cc: openembedded-core
Hi David,
On Tue, Feb 11, 2014 at 09:42:49AM +0100, David Nyström wrote:
> File: '/media/sdb5/poky-contrib/meta/lib/oe/image.py', lineno: 158,
> function: _write_script
> 0154: self.d.setVarFlag('img_creation_func', 'func', 1)
> 0155: self.d.setVarFlag('img_creation_func', 'fakeroot', 1)
> 0156:
> 0157: with open(script_name, "w+") as script:
> *** 0158: script.write("%s" % bb.build.shell_trap_code())
> 0159: script.write("export ROOTFS_SIZE=%d\n" %
> self._get_rootfs_size())
> 0160: bb.data.emit_func('img_creation_func', script,
> self.d)
> 0161: script.write("img_creation_func\n")
> 0162:
> Exception: AttributeError: 'module' object has no attribute
> 'shell_trap_code'
I guess you're missing this commit:
bitbake: build.py: create separate function for shell trap creation code
which is already in master.
laurentiu
>
> ERROR: Function failed: do_rootfs
> ERROR: Logfile of failure stored in: /media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.9994
> ERROR: Task 6 (/media/sdb5/poky-contrib/meta/recipes-core/images/core-image-minimal.bb,
> do_rootfs) failed with exit code '1'
>
>
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v5 00/53] Rootfs/image generation refactoring (cover letter only)
2014-02-11 8:50 ` Laurentiu Palcu
@ 2014-02-11 8:56 ` David Nyström
2014-02-11 9:36 ` David Nyström
0 siblings, 1 reply; 22+ messages in thread
From: David Nyström @ 2014-02-11 8:56 UTC (permalink / raw)
To: Laurentiu Palcu, David Nyström; +Cc: openembedded-core
On 2014-02-11 09:50, Laurentiu Palcu wrote:
> Hi David,
>
> On Tue, Feb 11, 2014 at 09:42:49AM +0100, David Nyström wrote:
>> File: '/media/sdb5/poky-contrib/meta/lib/oe/image.py', lineno: 158,
>> function: _write_script
>> 0154: self.d.setVarFlag('img_creation_func', 'func', 1)
>> 0155: self.d.setVarFlag('img_creation_func', 'fakeroot', 1)
>> 0156:
>> 0157: with open(script_name, "w+") as script:
>> *** 0158: script.write("%s" % bb.build.shell_trap_code())
>> 0159: script.write("export ROOTFS_SIZE=%d\n" %
>> self._get_rootfs_size())
>> 0160: bb.data.emit_func('img_creation_func', script,
>> self.d)
>> 0161: script.write("img_creation_func\n")
>> 0162:
>> Exception: AttributeError: 'module' object has no attribute
>> 'shell_trap_code'
> I guess you're missing this commit:
>
> bitbake: build.py: create separate function for shell trap creation code
>
> which is already in master.
Thanks,
David
> laurentiu
>
>
>>
>> ERROR: Function failed: do_rootfs
>> ERROR: Logfile of failure stored in: /media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.9994
>> ERROR: Task 6 (/media/sdb5/poky-contrib/meta/recipes-core/images/core-image-minimal.bb,
>> do_rootfs) failed with exit code '1'
>>
>>
>>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v5 00/53] Rootfs/image generation refactoring (cover letter only)
2014-02-11 8:56 ` David Nyström
@ 2014-02-11 9:36 ` David Nyström
0 siblings, 0 replies; 22+ messages in thread
From: David Nyström @ 2014-02-11 9:36 UTC (permalink / raw)
To: Laurentiu Palcu, David Nyström; +Cc: openembedded-core
On 2014-02-11 09:56, David Nyström wrote:
> On 2014-02-11 09:50, Laurentiu Palcu wrote:
>> Hi David,
>>
>> On Tue, Feb 11, 2014 at 09:42:49AM +0100, David Nyström wrote:
>>> File: '/media/sdb5/poky-contrib/meta/lib/oe/image.py', lineno: 158,
>>> function: _write_script
>>> 0154: self.d.setVarFlag('img_creation_func', 'func', 1)
>>> 0155: self.d.setVarFlag('img_creation_func', 'fakeroot', 1)
>>> 0156:
>>> 0157: with open(script_name, "w+") as script:
>>> *** 0158: script.write("%s" % bb.build.shell_trap_code())
>>> 0159: script.write("export ROOTFS_SIZE=%d\n" %
>>> self._get_rootfs_size())
>>> 0160: bb.data.emit_func('img_creation_func', script,
>>> self.d)
>>> 0161: script.write("img_creation_func\n")
>>> 0162:
>>> Exception: AttributeError: 'module' object has no attribute
>>> 'shell_trap_code'
>> I guess you're missing this commit:
>>
>> bitbake: build.py: create separate function for shell trap creation code
>>
>> which is already in master.
>
> Thanks,
>
> David
Tested-by: david.nystrom@enea.com
With the manifest stub patch I sent and v5, FEEDS functionality for IPK
is verified working again.
Only below is generated when building from FEEDS:
$ cat tmp/deploy/images/qemuppc/core-image-minimal-qemuppc.manifest
busybox-syslog ppc7400 1.21.1-r0
libc6 ppc7400 2.18-r0
busybox ppc7400 1.21.1-r0
busybox-udhcpc ppc7400 1.21.1-r0
modutils-initscripts ppc7400 1.0-r6
libuuid1 ppc7400 2.23.2-r0
base-files qemuppc 3.0.14-r73
sysvinit-inittab qemuppc 2.88dsf-r9
initscripts ppc7400 1.0-r141
update-alternatives-cworth ppc7400 1:0.1.8+svnr653-r13.0
libblkid1 ppc7400 2.23.2-r0
udev-utils ppc7400 182-r7
libkmod2 ppc7400 14+git0+3b38c7fcb5-r0
base-passwd ppc7400 3.5.26-r2
busybox-hwclock ppc7400 1.21.1-r0
packagegroup-core-boot qemuppc 1.0-r11
udev ppc7400 182-r7
sysvinit-pidof ppc7400 2.88dsf-r11
update-rc.d all 0.7-r5
netbase ppc7400 1:5.1-r0
udev-cache ppc7400 182-r7
run-postinsts ppc7400 1.0-r9
init-ifupdown qemuppc 1.0-r3
sysvinit ppc7400 2.88dsf-r11
Br,
David
>> laurentiu
>>
>>
>>>
>>> ERROR: Function failed: do_rootfs
>>> ERROR: Logfile of failure stored in:
>>> /media/sdb5/bld/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/temp/log.do_rootfs.9994
>>>
>>> ERROR: Task 6
>>> (/media/sdb5/poky-contrib/meta/recipes-core/images/core-image-minimal.bb,
>>>
>>> do_rootfs) failed with exit code '1'
>>>
>>>
>>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v5 00/53] Rootfs/image generation refactoring (cover letter only)
2014-02-10 12:15 ` [PATCH v5 00/53] " Laurentiu Palcu
2014-02-11 8:42 ` David Nyström
@ 2014-02-11 14:33 ` David Nyström
2014-02-11 14:50 ` David Nyström
1 sibling, 1 reply; 22+ messages in thread
From: David Nyström @ 2014-02-11 14:33 UTC (permalink / raw)
To: Laurentiu Palcu, openembedded-core
On 2014-02-10 13:15, Laurentiu Palcu wrote:
> V5:
> * dump command output in case errors occur;
>
rm -fr tmp/deploy/qemuppc/images/*
bitbake -c cleanall core-image-minimal; bitbake core-image-minimal
Above does not seem to regenerate the image ?
Br,
David
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v5 00/53] Rootfs/image generation refactoring (cover letter only)
2014-02-11 14:33 ` David Nyström
@ 2014-02-11 14:50 ` David Nyström
0 siblings, 0 replies; 22+ messages in thread
From: David Nyström @ 2014-02-11 14:50 UTC (permalink / raw)
To: David Nyström, Laurentiu Palcu, openembedded-core
On 2014-02-11 15:33, David Nyström wrote:
> On 2014-02-10 13:15, Laurentiu Palcu wrote:
>> V5:
>> * dump command output in case errors occur;
>>
>
> rm -fr tmp/deploy/qemuppc/images/*
> bitbake -c cleanall core-image-minimal; bitbake core-image-minimal
>
> Above does not seem to regenerate the image ?
Sorry, Ignore this.
My mistake.
> Br,
> David
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2014-02-11 14:50 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-03 11:38 [PATCH 00/49] Rootfs/image generation refactoring (cover letter only) Laurentiu Palcu
2014-02-03 11:54 ` Laurentiu Palcu
2014-02-03 14:32 ` David Nyström
2014-02-03 16:47 ` Laurentiu Palcu
2014-02-04 14:08 ` David Nyström
2014-02-04 14:49 ` David Nyström
2014-02-04 15:34 ` Laurentiu Palcu
2014-02-04 15:40 ` David Nyström
2014-02-05 10:07 ` [PATCH v2 00/50] " Laurentiu Palcu
2014-02-05 10:58 ` David Nystrom
2014-02-05 12:09 ` David Nyström
2014-02-05 14:43 ` Laurentiu Palcu
2014-02-05 18:00 ` David Nystrom
2014-02-05 15:05 ` [PATCH v3 00/51] " Laurentiu Palcu
2014-02-10 7:53 ` [PATCH v4 00/52] " Laurentiu Palcu
2014-02-10 12:15 ` [PATCH v5 00/53] " Laurentiu Palcu
2014-02-11 8:42 ` David Nyström
2014-02-11 8:50 ` Laurentiu Palcu
2014-02-11 8:56 ` David Nyström
2014-02-11 9:36 ` David Nyström
2014-02-11 14:33 ` David Nyström
2014-02-11 14:50 ` David Nyström
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox