* [PATCH 0/1] wic: Add wic support to generate rootfs image for uboot
@ 2014-06-10 13:41 Adrian Calianu
2014-06-10 13:41 ` [PATCH 1/1] " Adrian Calianu
0 siblings, 1 reply; 11+ messages in thread
From: Adrian Calianu @ 2014-06-10 13:41 UTC (permalink / raw)
To: openembedded-core
Hi,
We tried to extend existing wic tool to generates rootfs images for U-Boot.
This will be possible from cooked mode(existing build folder) and from raw
mode where you have to provide a list of packages to be installed and
a native rootfs with necessary tools.
Some prerequisites are required for this new build setup in wic.conf file.
As package manager only opkg and only web paths(repourl) are supported by now.
../poky/scripts/lib/image/config/wic.conf
[create]
arch=target_arch (example: powerpc, arm, x86)
pkgmgr=opkg
repourl=http://example.distro/p2020rdb/ipk/all http://example.distro/p2020rdb/ipk/p2020rdb http://example.distro/p2020rdb/ipk/ppce500v2
1) Build rootfs image from an existing bitbake build:
wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core-image-minimal -o .../output
2) Build rootfs image from an existing rootfs and native_sysroot:
wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r .../path_to_rootfs -n .../path_to_native_sysroot -o .../output
3) Build rootfs image only from a package list (on wks file) similar with IMAGE_INSTALL from bitbake.
Package dependencies will be installed by default by the package manager.
Add the list of packages to be installed as rootfs on ../poky/scripts/lib/image/canned-wks/uboot.wks:
%packages
packagegroup-core-boot
pramfs-init
run-postinsts
packagegroup-core-ssh-dropbear
%end
Generate rootfs image:
wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n .../path_to_native_sysroot -o .../output
Adrian Calianu (1):
Add wic support to generate rootfs image for uboot
scripts/lib/image/canned-wks/uboot.wks | 17 +
scripts/lib/image/config/wic.conf | 4 +
scripts/lib/image/engine.py | 7 +-
scripts/lib/mic/imager/direct.py | 4 +
.../lib/mic/kickstart/custom_commands/partition.py | 128 ++++
scripts/lib/mic/plugin.py | 11 +-
scripts/lib/mic/pluginbase.py | 9 +
scripts/lib/mic/plugins/source/uboot.py | 173 +++++
scripts/lib/mic/utils/oe/package_manager.py | 810 ++++++++++++++++++++
scripts/wic | 88 ++-
10 files changed, 1219 insertions(+), 32 deletions(-)
create mode 100644 scripts/lib/image/canned-wks/uboot.wks
create mode 100644 scripts/lib/mic/plugins/source/uboot.py
create mode 100644 scripts/lib/mic/utils/oe/package_manager.py
--
1.7.10.4
^ permalink raw reply [flat|nested] 11+ messages in thread* [PATCH 1/1] Add wic support to generate rootfs image for uboot 2014-06-10 13:41 [PATCH 0/1] wic: Add wic support to generate rootfs image for uboot Adrian Calianu @ 2014-06-10 13:41 ` Adrian Calianu 2014-06-12 2:49 ` Saul Wold 0 siblings, 1 reply; 11+ messages in thread From: Adrian Calianu @ 2014-06-10 13:41 UTC (permalink / raw) To: openembedded-core Add support in wic to build ramdisk uboot images from list of packages or from an existing rootfs folder. Some prerequisites are required for this new build setup: ../poky/scripts/lib/image/config/wic.conf [create] arch=target_arch (example: powerpc, arm, x86) pkgmgr=opkg repourl=http://example.distro/p2020rdb/ipk/all http://example.distro/p2020rdb/ipk/p2020rdb http://example.distro/p2020rdb/ipk/ppce500v2 1) Build an rootfs image from an existing bitbake build: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core-image-minimal -o .../output 2) Build an rootfs image from an existing rootfs and native_sysroot: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r .../path_to_rootfs -n .../path_to_native_sysroot -o .../output 3) Build an rootfs image only from a package list (on wks file): Add the package list to be installed as rootfs on ../poky/scripts/lib/image/canned-wks/uboot.wks: %packages packagegroup-core-boot pramfs-init run-postinsts packagegroup-core-ssh-dropbear %end Generate rootfs image: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n .../path_to_native_sysroot -o .../output Signed-off-by: Adrian Calianu <adrian.calianu@enea.com> --- scripts/lib/image/canned-wks/uboot.wks | 17 + scripts/lib/image/config/wic.conf | 4 + scripts/lib/image/engine.py | 7 +- scripts/lib/mic/imager/direct.py | 4 + .../lib/mic/kickstart/custom_commands/partition.py | 128 ++++ scripts/lib/mic/plugin.py | 11 +- scripts/lib/mic/pluginbase.py | 9 + scripts/lib/mic/plugins/source/uboot.py | 173 +++++ scripts/lib/mic/utils/oe/package_manager.py | 810 ++++++++++++++++++++ scripts/wic | 88 ++- 10 files changed, 1219 insertions(+), 32 deletions(-) create mode 100644 scripts/lib/image/canned-wks/uboot.wks create mode 100644 scripts/lib/mic/plugins/source/uboot.py create mode 100644 scripts/lib/mic/utils/oe/package_manager.py diff --git a/scripts/lib/image/canned-wks/uboot.wks b/scripts/lib/image/canned-wks/uboot.wks new file mode 100644 index 0000000..7de0572 --- /dev/null +++ b/scripts/lib/image/canned-wks/uboot.wks @@ -0,0 +1,17 @@ +# short-description: . Create a ramdisk image for U-Boot +# long-description: Creates a ramdisk image for U-Boot that user +# can directly load it into ram through tftp +# +# part - is a wic command that drive the process of generating a valid file system +# - --source=uboot : wic plugin that generates a ramdisk image for U-Boot +# - --fstype=ext2 : file system type( ext2 / ext3 / ext 4) +# +# %packages %end - option to provide a list of packages that will be installed +# into rootfs. All packages dependencies will be installed by +# package manager(default opkg). + + +part / --source=uboot --fstype=ext2 --label imageName --align 1024 + +%packages +%end diff --git a/scripts/lib/image/config/wic.conf b/scripts/lib/image/config/wic.conf index e96d6ae..2a2750b 100644 --- a/scripts/lib/image/config/wic.conf +++ b/scripts/lib/image/config/wic.conf @@ -4,4 +4,8 @@ distro_name = OpenEmbedded [create] ; settings for create subcommand +; repourl=http://linux.com/ipk/all http://linux.com/ipk/target http://linux.com/ipk/arch +arch=powerpc +pkgmgr=opkg runtime=native +install_pkgs=source diff --git a/scripts/lib/image/engine.py b/scripts/lib/image/engine.py index 3bda1bf..1256236 100644 --- a/scripts/lib/image/engine.py +++ b/scripts/lib/image/engine.py @@ -96,9 +96,10 @@ def build_canned_image_list(dl): layers_path = get_bitbake_var("BBLAYERS") canned_wks_layer_dirs = [] - for layer_path in layers_path.split(): - path = os.path.join(layer_path, SCRIPTS_CANNED_IMAGE_DIR) - canned_wks_layer_dirs.append(path) + if layers_path is not None: + for layer_path in layers_path.split(): + path = os.path.join(layer_path, SCRIPTS_CANNED_IMAGE_DIR) + canned_wks_layer_dirs.append(path) path = os.path.join(dl, CANNED_IMAGE_DIR) canned_wks_layer_dirs.append(path) diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py index 2cf4c8d..fef9d0e 100644 --- a/scripts/lib/mic/imager/direct.py +++ b/scripts/lib/mic/imager/direct.py @@ -262,6 +262,10 @@ class DirectImageCreator(BaseImageCreator): # when/if we need to actually do package selection we # should modify things to use those objects, but for now # we can avoid that. + + p.install_pkgs(self, self.workdir, self.oe_builddir, self.rootfs_dir, + self.bootimg_dir, self.kernel_dir, self.native_sysroot) + p.prepare(self, self.workdir, self.oe_builddir, self.rootfs_dir, self.bootimg_dir, self.kernel_dir, self.native_sysroot) diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py index 6b575c0..450d2d4 100644 --- a/scripts/lib/mic/kickstart/custom_commands/partition.py +++ b/scripts/lib/mic/kickstart/custom_commands/partition.py @@ -31,7 +31,11 @@ from mic.utils.oe.misc import * from mic.kickstart.custom_commands import * from mic.plugin import pluginmgr +import os +from mic.utils.oe.package_manager import * + partition_methods = { + "do_install_pkgs":None, "do_stage_partition":None, "do_prepare_partition":None, "do_configure_partition":None, @@ -115,6 +119,102 @@ class Wic_PartData(Mic_PartData): else: return 0 + def install_pkgs(self, creator, cr_workdir, oe_builddir, rootfs_dir, + bootimg_dir, kernel_dir, native_sysroot): + """ + Prepare content for individual partitions, installing packages. + """ + + if not self.source: + return + + self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods) + self._source_methods["do_install_pkgs"](self, creator, + cr_workdir, + oe_builddir, + rootfs_dir, + bootimg_dir, + kernel_dir, + native_sysroot) + + def install_pkgs_ipk(self, cr_workdir, oe_builddir, rootfs_dir, + native_sysroot, packages, repourl): + """ + Install packages specified into wks file using opkg package manager. + This method is dependend on bb module. + """ + + gVar = {} + gVar["DEPLOY_DIR_IPK"] = os.path.join(oe_builddir, "tmp/deploy/ipk") + + # Run postinstall scripts even in offline mode + # Use the arch priority package rather than higher version one if more than one candidate is found. + #d.setVar("OPKG_ARGS", "--force_postinstall --prefer-arch-to-version") + gVar["OPKG_ARGS"] = "--force_postinstall" + + # OPKG path relative to /output_path + gVar["OPKGLIBDIR"] = "var/lib" + + source_url = repourl.split() + + # Generate feed uri's names, it doesn't seem to matter what name they have + feed_uris = "" + cnt = 0 + archs = "" + for url in source_url: + feed_uris += "cl_def_feed%d##%s\n" % (cnt, url) + cnt += 1 + head, tail = os.path.split(url) + archs += " " + tail + + # IPK_FEED_URIS with special formating defines the URI's used as source for packages + gVar['IPK_FEED_URIS'] = feed_uris + + gVar['BUILD_IMAGES_FROM_FEEDS'] = "1" + + # We need to provide sysroot for utilities + gVar['STAGING_DIR_NATIVE'] = native_sysroot + + # Set WORKDIR for output + gVar['WORKDIR'] = cr_workdir + + # Set TMPDIR for output + gVar['TMPDIR'] = os.path.join(cr_workdir, "tmp") + + if 'ROOTFS_DIR' in rootfs_dir: + target_dir = rootfs_dir['ROOTFS_DIR'] + elif os.path.isdir(rootfs_dir): + target_dir = rootfs_dir + else: + msg = "Couldn't find --rootfs-dir=%s connection" + msg += " or it is not a valid path, exiting" + msger.error(msg % rootfs_dir) + + # Need native sysroot /usr/bin/ for opkg-cl + # chnage PATH var to avoid issues with host tools + defpath = os.environ['PATH'] + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" + + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % target_dir + pseudo += "export PSEUDO_PASSWD=%s;" % target_dir + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "%s/usr/bin/pseudo " % native_sysroot + + pm = WicOpkgPM(gVar, + target_dir, + 'opkg.conf', + archs, + pseudo, + native_sysroot) + + pm.update() + + pm.install(packages) + + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" + + def prepare(self, cr, cr_workdir, oe_builddir, rootfs_dir, bootimg_dir, kernel_dir, native_sysroot): """ @@ -225,6 +325,34 @@ class Wic_PartData(Mic_PartData): return 0 + def prepare_for_uboot(self, arch, cr_workdir, oe_builddir, rootfs_dir, + native_sysroot): + """ + Generates u-boot image from source_file( ext2/3/4 ) + + """ + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "%s/usr/bin/pseudo " % native_sysroot + + # 1) compress image + rootfs = self.source_file + rootfs_gzip = "%s.gz" % rootfs + gzip_cmd = "gzip -f -9 -c %s > %s" % (rootfs, rootfs_gzip) + rc, out = exec_native_cmd(pseudo + gzip_cmd, native_sysroot) + + # 2) image for U-Boot + rootfs_uboot = "%s.u-boot" % rootfs_gzip + mkimage_cmd = "mkimage -A %s -O linux -T ramdisk -C gzip -n %s -d %s %s" % \ + (arch, self.label, rootfs_gzip, rootfs_uboot) + rc, out = exec_native_cmd(pseudo + mkimage_cmd, native_sysroot) + + msger.info("\n\n\tThe new U-Boot ramdisk image can be found here:\n\t\t%s\n\n" % rootfs_uboot) + + return 0 + def prepare_rootfs_btrfs(self, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ diff --git a/scripts/lib/mic/plugin.py b/scripts/lib/mic/plugin.py index bec33d6..585fd6d 100644 --- a/scripts/lib/mic/plugin.py +++ b/scripts/lib/mic/plugin.py @@ -53,12 +53,13 @@ class PluginMgr(object): self.layers_path = get_bitbake_var("BBLAYERS") layer_dirs = [] - for layer_path in self.layers_path.split(): - path = os.path.join(layer_path, SCRIPTS_PLUGIN_DIR, ptype) - layer_dirs.append(path) + if self.layers_path is not None: + for layer_path in self.layers_path.split(): + path = os.path.join(layer_path, SCRIPTS_PLUGIN_DIR, ptype) + layer_dirs.append(path) - path = os.path.join(dl, ptype) - layer_dirs.append(path) + path = os.path.join(dl, ptype) + layer_dirs.append(path) return layer_dirs diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py index 9cf4c62..881d996 100644 --- a/scripts/lib/mic/pluginbase.py +++ b/scripts/lib/mic/pluginbase.py @@ -89,6 +89,15 @@ class SourcePlugin(_Plugin): """ @classmethod + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, rootfs_dir, + bootimg_dir, kernel_dir, native_sysroot): + """ + Called before partitions have been prepared and assembled into a + disk image. Install packages into rootfs + """ + msger.debug("SourcePlugin: do_install_pkgs: part %s" % part) + + @classmethod def do_install_disk(self, disk, disk_name, cr, workdir, oe_builddir, bootimg_dir, kernel_dir, native_sysroot): """ diff --git a/scripts/lib/mic/plugins/source/uboot.py b/scripts/lib/mic/plugins/source/uboot.py new file mode 100644 index 0000000..57cb3cf --- /dev/null +++ b/scripts/lib/mic/plugins/source/uboot.py @@ -0,0 +1,173 @@ +# ex:ts=4:sw=4:sts=4:et +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- +# +# Copyright (c) 2014, Enea AB. +# All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# DESCRIPTION +# This implements the 'uboot' source plugin class for 'wic' +# +# AUTHORS +# Adrian Calianu <adrian.calianu (at] enea.com> +# + +import os +import shutil +import re +import tempfile + +from mic import kickstart, chroot, msger +from mic.utils import misc, fs_related, errors, runner, cmdln +from mic.conf import configmgr +from mic.plugin import pluginmgr +from mic.utils.partitionedfs import PartitionedMount +import mic.imager.direct as direct +from mic.pluginbase import SourcePlugin +from mic.utils.oe.misc import * +from mic.imager.direct import DirectImageCreator + +def create_local_rootfs(part, creator, cr_workdir, krootfs_dir, native_sysroot): + # In order to have a full control over rootfs we will make a local copy under workdir + # and change rootfs_dir to new location. + # In this way we can install more than one ROOTFS_DIRs and/or use + # an empty rootfs to install packages, so a rootfs could be generated only from pkgs + # TBD: create workdir/rootfs ; copy rootfs-> workdir/rootfs; set rootfs=workdir/rootfs + + cr_workdir = os.path.abspath(cr_workdir) + new_rootfs_dir = "%s/rootfs_%s" % (cr_workdir, creator.name) + + rootfs_exists = 1 + if part.rootfs is None: + if not 'ROOTFS_DIR' in krootfs_dir: + msg = "Couldn't find --rootfs-dir, exiting, " + msger.info(msg) + rootfs_exists = 0 + rootfs_dir = krootfs_dir['ROOTFS_DIR'] + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir + else: + if part.rootfs in krootfs_dir: + rootfs_dir = krootfs_dir[part.rootfs] + creator.rootfs_dir[part.rootfs] = new_rootfs_dir + elif os.path.isdir(part.rootfs): + rootfs_dir = part.rootfs + part.rootfs = new_rootfs_dir + else: + msg = "Couldn't find --rootfs-dir=%s connection" + msg += " or it is not a valid path, exiting" + msger.info(msg % part.rootfs) + rootfs_exists = 0 + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir + + pseudox = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudox += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % new_rootfs_dir + pseudox += "export PSEUDO_PASSWD=%s;" % new_rootfs_dir + pseudox += "export PSEUDO_NOSYMLINKEXP=1;" + pseudox += "%s/usr/bin/pseudo " % native_sysroot + + mkdir_cmd = "mkdir %s" % (new_rootfs_dir) + # rc, out = exec_native_cmd(pseudox + mkdir_cmd, native_sysroot) + rc, out = exec_cmd(mkdir_cmd, True) + + if rootfs_exists == 1 and os.path.isdir(rootfs_dir): + defpath = os.environ['PATH'] + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" + + rootfs_dir = os.path.abspath(rootfs_dir) + + pseudoc = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudoc += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir + pseudoc += "export PSEUDO_PASSWD=%s;" % rootfs_dir + pseudoc += "export PSEUDO_NOSYMLINKEXP=1;" + pseudoc += "%s/usr/bin/pseudo " % native_sysroot + + tarc_cmd = "tar cvpf %s/rootfs.tar -C %s ." % (cr_workdir, rootfs_dir) + rc, out = exec_native_cmd(pseudoc + tarc_cmd, native_sysroot) + + tarx_cmd = "tar xpvf %s/rootfs.tar -C %s" % (cr_workdir, new_rootfs_dir) + rc, out = exec_native_cmd(pseudox + tarx_cmd, native_sysroot) + + rm_cmd = "rm %s/rootfs.tar" % cr_workdir + rc, out = exec_cmd(rm_cmd, True) + + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" + + return new_rootfs_dir + +class UBootPlugin(SourcePlugin): + name = 'uboot' + + @classmethod + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, krootfs_dir, + bootimg_dir, kernel_dir, native_sysroot): + """ + Called before all partitions have been prepared and assembled into a + disk image. Intall packages based on wic configuration. + """ + + # set new rootfs_dir + rootfs_dir = create_local_rootfs(part, creator, cr_workdir, krootfs_dir, native_sysroot) + + # wks file parsing + packages = kickstart.get_packages(creator.ks) + + # wic.conf file parsing = found under 'creator' + local_pkgs_path = creator._local_pkgs_path + repourl = creator.repourl + pkgmgr = creator.pkgmgr_name + + # install packages + if packages and pkgmgr in ["opkg"]: + if len(repourl) > 0 : + part.install_pkgs_ipk(cr_workdir, oe_builddir, rootfs_dir, native_sysroot, + packages, repourl) + else: + msger.error("No packages repository provided in wic.conf") + + @classmethod + def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir, + kernel_dir, krootfs_dir, native_sysroot): + """ + Called to do the actual content population for a partition i.e. it + 'prepares' the partition to be incorporated into the image. + In this case, prepare content for legacy bios boot partition. + """ + if part.rootfs is None: + if not 'ROOTFS_DIR' in krootfs_dir: + msg = "Couldn't find --rootfs-dir, exiting" + msger.error(msg) + rootfs_dir = krootfs_dir['ROOTFS_DIR'] + else: + if part.rootfs in krootfs_dir: + rootfs_dir = krootfs_dir[part.rootfs] + elif os.path.isdir(part.rootfs): + rootfs_dir = part.rootfs + else: + msg = "Couldn't find --rootfs-dir=%s connection" + msg += " or it is not a valid path, exiting" + msger.error(msg % part.rootfs) + + part.set_rootfs(rootfs_dir) + + # change partition label wich will reflect into the final rootfs image name + part.label = "%s_%s" % (part.label, cr.name) + + defpath = os.environ['PATH'] + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" + + part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot) + part.prepare_for_uboot(cr.target_arch,cr_workdir, oe_builddir, rootfs_dir, native_sysroot) + + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" diff --git a/scripts/lib/mic/utils/oe/package_manager.py b/scripts/lib/mic/utils/oe/package_manager.py new file mode 100644 index 0000000..92ce98e --- /dev/null +++ b/scripts/lib/mic/utils/oe/package_manager.py @@ -0,0 +1,810 @@ +# ex:ts=4:sw=4:sts=4:et +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- +# +# Copyright (c) 2014, Enea AB. +# All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# DESCRIPTION +# This implements the opkg package manager wrapper as a combination of +# meta/lib/oe/package_manager.py and bitbake/lib/bb/utils.py files and +# adaptation of those files to 'wic'. +# +# AUTHORS +# Adrian Calianu <adrian.calianu (at] enea.com> +# +# This file incorporates work covered by the following copyright and +# permission notice: +# +# meta/COPYING.GPLv2 (GPLv2) +# meta/COPYING.MIT (MIT) +# +# Copyright (C) 2004 Michael Lauer +# +# Permission to use, copy, modify, and/or distribute this software +# for any purpose with or without fee is hereby granted, provided +# that the above copyright notice and this permission notice appear +# in all copies. +# +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL +# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE +# AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR +# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS +# OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, +# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN +# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + +from abc import ABCMeta, abstractmethod +import os +import glob +import subprocess +import shutil +import multiprocessing +import re +import errno +import fcntl + +from mic.utils.oe.misc import * +from mic import msger + +def mkdirhier(directory): + """Create a directory like 'mkdir -p', but does not complain if + directory already exists like os.makedirs + """ + + try: + os.makedirs(directory) + except OSError as e: + if e.errno != errno.EEXIST: + raise e + +def remove(path, recurse=False): + """Equivalent to rm -f or rm -rf""" + if not path: + return + if recurse: + # shutil.rmtree(name) would be ideal but its too slow + subprocess.call(['rm', '-rf'] + glob.glob(path)) + return + for name in glob.glob(path): + try: + os.unlink(name) + except OSError as exc: + if exc.errno != errno.ENOENT: + raise + +def lockfile(name, shared=False, retry=True): + """ + Use the file fn as a lock file, return when the lock has been acquired. + Returns a variable to pass to unlockfile(). + """ + dirname = os.path.dirname(name) + mkdirhier(dirname) + + if not os.access(dirname, os.W_OK): + logger.error("Unable to acquire lock '%s', directory is not writable", + name) + sys.exit(1) + + op = fcntl.LOCK_EX + if shared: + op = fcntl.LOCK_SH + if not retry: + op = op | fcntl.LOCK_NB + + while True: + # If we leave the lockfiles lying around there is no problem + # but we should clean up after ourselves. This gives potential + # for races though. To work around this, when we acquire the lock + # we check the file we locked was still the lock file on disk. + # by comparing inode numbers. If they don't match or the lockfile + # no longer exists, we start again. + + # This implementation is unfair since the last person to request the + # lock is the most likely to win it. + + try: + lf = open(name, 'a+') + fileno = lf.fileno() + fcntl.flock(fileno, op) + statinfo = os.fstat(fileno) + if os.path.exists(lf.name): + statinfo2 = os.stat(lf.name) + if statinfo.st_ino == statinfo2.st_ino: + return lf + lf.close() + except Exception: + try: + lf.close() + except Exception: + pass + pass + if not retry: + return None + +def unlockfile(lf): + """ + Unlock a file locked using lockfile() + """ + try: + # If we had a shared lock, we need to promote to exclusive before + # removing the lockfile. Attempt this, ignore failures. + fcntl.flock(lf.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB) + os.unlink(lf.name) + except (IOError, OSError): + pass + fcntl.flock(lf.fileno(), fcntl.LOCK_UN) + lf.close() + +def which(path, item, direction = 0, history = False): + """ + Locate a file in a PATH + """ + + hist = [] + paths = (path or "").split(':') + if direction != 0: + paths.reverse() + + for p in paths: + next = os.path.join(p, item) + hist.append(next) + if os.path.exists(next): + if not os.path.isabs(next): + next = os.path.abspath(next) + if history: + return next, hist + return next + + if history: + return "", hist + return "" + + + +# this can be used by all PM backends to create the index files in parallel +def wic_create_index(arg): + index_cmd = arg + + try: + msger.info("Executing '%s' ..." % index_cmd) + subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True) + except subprocess.CalledProcessError as e: + return("Index creation command '%s' failed with return code %d:\n%s" % + (e.cmd, e.returncode, e.output)) + + return None + + +class WicIndexer(object): + __metaclass__ = ABCMeta + + def __init__(self, d, deploy_dir): + self.d = d + self.deploy_dir = deploy_dir + + @abstractmethod + def write_index(self): + pass + +class WicOpkgIndexer(WicIndexer): + def write_index(self): + arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS", + "SDK_PACKAGE_ARCHS", + "MULTILIB_ARCHS"] + + opkg_index_cmd = which(os.getenv('PATH'), "opkg-make-index") + + if not os.path.exists(os.path.join(self.deploy_dir, "Packages")): + open(os.path.join(self.deploy_dir, "Packages"), "w").close() + + index_cmds = [] + for arch_var in arch_vars: + if self.d.has_key(arch_var): + archs = self.d[arch_var] + else: + archs = None + + if archs is None: + continue + + for arch in archs.split(): + pkgs_dir = os.path.join(self.deploy_dir, arch) + pkgs_file = os.path.join(pkgs_dir, "Packages") + + if not os.path.isdir(pkgs_dir): + continue + + if not os.path.exists(pkgs_file): + open(pkgs_file, "w").close() + + index_cmds.append('%s -r %s -p %s -m %s' % + (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir)) + + if len(index_cmds) == 0: + msger.info("There are no packages in %s!" % self.deploy_dir) + return + + nproc = multiprocessing.cpu_count() + pool = multiprocessing.Pool(nproc) + results = list(pool.imap(wic_create_index, index_cmds)) + pool.close() + pool.join() + + for result in results: + if result is not None: + return(result) + +class WicPkgsList(object): + __metaclass__ = ABCMeta + + def __init__(self, d, rootfs_dir): + self.d = d + self.rootfs_dir = rootfs_dir + + @abstractmethod + def list(self, format=None): + pass + + +class WicOpkgPkgsList(WicPkgsList): + def __init__(self, d, rootfs_dir, config_file): + super(WicOpkgPkgsList, self).__init__(d, rootfs_dir) + + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") + self.opkg_args = "-f %s -o %s " % (config_file, rootfs_dir) + if self.d.has_key("OPKG_ARGS"): + self.opkg_args += self.d["OPKG_ARGS"] + + def list(self, format=None): + opkg_query_cmd = which(os.getenv('PATH'), "opkg-query-helper.py") + + if format == "arch": + cmd = "%s %s status | %s -a" % \ + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) + elif format == "file": + cmd = "%s %s status | %s -f" % \ + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) + elif format == "ver": + cmd = "%s %s status | %s -v" % \ + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) + elif format == "deps": + cmd = "%s %s status | %s" % \ + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) + else: + cmd = "%s %s list_installed | cut -d' ' -f1" % \ + (self.opkg_cmd, self.opkg_args) + + try: + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip() + except subprocess.CalledProcessError as e: + msger.error("Cannot get the installed packages list. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + if output and format == "file": + tmp_output = "" + for line in output.split('\n'): + pkg, pkg_file, pkg_arch = line.split() + full_path = os.path.join(self.rootfs_dir, pkg_arch, pkg_file) + if os.path.exists(full_path): + tmp_output += "%s %s %s\n" % (pkg, full_path, pkg_arch) + else: + tmp_output += "%s %s %s\n" % (pkg, pkg_file, pkg_arch) + + output = tmp_output + + return output + + +class WicPackageManager(object): + """ + This is an abstract class. Do not instantiate this directly. + """ + __metaclass__ = ABCMeta + + def __init__(self, d, pseudo, native_sysroot): + self.d = d + self.deploy_dir = None + self.deploy_lock = None + if self.d.has_key('PACKAGE_FEED_URIS'): + self.feed_uris = self.d['PACKAGE_FEED_URIS'] + else: + self.feed_uris = "" + self.pseudo = pseudo + self.native_sysroot = native_sysroot + + """ + Update the package manager package database. + """ + @abstractmethod + def update(self): + pass + + """ + Install a list of packages. 'pkgs' is a list object. If 'attempt_only' is + True, installation failures are ignored. + """ + @abstractmethod + def install(self, pkgs, attempt_only=False): + pass + + """ + Remove a list of packages. 'pkgs' is a list object. If 'with_dependencies' + is False, the any dependencies are left in place. + """ + @abstractmethod + def remove(self, pkgs, with_dependencies=True): + pass + + """ + This function creates the index files + """ + @abstractmethod + def write_index(self): + pass + + @abstractmethod + def remove_packaging_data(self): + pass + + @abstractmethod + def list_installed(self, format=None): + pass + + @abstractmethod + def insert_feeds_uris(self): + pass + + """ + Install complementary packages based upon the list of currently installed + packages e.g. locales, *-dev, *-dbg, etc. This will only attempt to install + these packages, if they don't exist then no error will occur. Note: every + backend needs to call this function explicitly after the normal package + installation + """ + def install_complementary(self, globs=None): + # we need to write the list of installed packages to a file because the + # oe-pkgdata-util reads it from a file + if self.d.has_key('WORKDIR'): + installed_pkgs_file = os.path.join(self.d['WORKDIR'], + "installed_pkgs.txt") + else: + msger.error("No WORKDIR provided!") + + with open(installed_pkgs_file, "w+") as installed_pkgs: + installed_pkgs.write(self.list_installed("arch")) + + if globs is None: + if self.d.has_key('IMAGE_INSTALL_COMPLEMENTARY'): + globs = self.d['IMAGE_INSTALL_COMPLEMENTARY'] + split_linguas = set() + + if self.d.has_key('IMAGE_LINGUAS'): + for translation in self.d['IMAGE_LINGUAS'].split(): + split_linguas.add(translation) + split_linguas.add(translation.split('-')[0]) + + split_linguas = sorted(split_linguas) + + for lang in split_linguas: + globs += " *-locale-%s" % lang + + if globs is None: + return + + if not self.d.has_key('PKGDATA_DIR'): + msger.error("No PKGDATA_DIR provided!") + + cmd = [which(os.getenv('PATH'), "oe-pkgdata-util"), + "glob", self.d['PKGDATA_DIR'], installed_pkgs_file, + globs] + + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) + if rc != 0: + msger.error("Could not compute complementary packages list. Command " + "'%s' returned %d" % + (' '.join(cmd), rc)) + + self.install(out.split(), attempt_only=True) + + + def deploy_dir_lock(self): + if self.deploy_dir is None: + raise RuntimeError("deploy_dir is not set!") + + lock_file_name = os.path.join(self.deploy_dir, "deploy.lock") + + self.deploy_lock = lockfile(lock_file_name) + + def deploy_dir_unlock(self): + if self.deploy_lock is None: + return + + unlockfile(self.deploy_lock) + + self.deploy_lock = None + + +class WicOpkgPM(WicPackageManager): + def __init__(self, d, target_rootfs, config_file, archs, pseudo, native_sysroot, task_name='target'): + super(WicOpkgPM, self).__init__(d, pseudo, native_sysroot) + + self.target_rootfs = target_rootfs + self.config_file = config_file + self.pkg_archs = archs + self.task_name = task_name + + if self.d.has_key("DEPLOY_DIR_IPK"): + self.deploy_dir = self.d["DEPLOY_DIR_IPK"] + + self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock") + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") + self.opkg_args = "-f %s -o %s " % (self.config_file, target_rootfs) + if self.d.has_key("OPKG_ARGS"): + self.opkg_args += self.d["OPKG_ARGS"] + + if self.d.has_key('OPKGLIBDIR'): + opkg_lib_dir = self.d['OPKGLIBDIR'] + else: + opkg_lib_dir = "" + + if opkg_lib_dir[0] == "/": + opkg_lib_dir = opkg_lib_dir[1:] + + self.opkg_dir = os.path.join(target_rootfs, opkg_lib_dir, "opkg") + + mkdirhier(self.opkg_dir) + + if self.d.has_key("TMPDIR"): + tmp_dir = self.d["TMPDIR"] + else: + tmp_dir = "" + + self.saved_opkg_dir = '%s/saved/%s' % (tmp_dir, self.task_name) + if not os.path.exists('%s/saved' % tmp_dir): + mkdirhier('%s/saved' % tmp_dir) + + if self.d.has_key('BUILD_IMAGES_FROM_FEEDS') and self.d['BUILD_IMAGES_FROM_FEEDS'] != "1": + self._create_config() + else: + self._create_custom_config() + + self.indexer = WicOpkgIndexer(self.d, self.deploy_dir) + + """ + This function will change a package's status in /var/lib/opkg/status file. + If 'packages' is None then the new_status will be applied to all + packages + """ + def mark_packages(self, status_tag, packages=None): + status_file = os.path.join(self.opkg_dir, "status") + + with open(status_file, "r") as sf: + with open(status_file + ".tmp", "w+") as tmp_sf: + if packages is None: + tmp_sf.write(re.sub(r"Package: (.*?)\n((?:[^\n]+\n)*?)Status: (.*)(?:unpacked|installed)", + r"Package: \1\n\2Status: \3%s" % status_tag, + sf.read())) + else: + if type(packages).__name__ != "list": + raise TypeError("'packages' should be a list object") + + status = sf.read() + for pkg in packages: + status = re.sub(r"Package: %s\n((?:[^\n]+\n)*?)Status: (.*)(?:unpacked|installed)" % pkg, + r"Package: %s\n\1Status: \2%s" % (pkg, status_tag), + status) + + tmp_sf.write(status) + + os.rename(status_file + ".tmp", status_file) + + def _create_custom_config(self): + msger.info("Building from feeds activated!") + + with open(self.config_file, "w+") as config_file: + priority = 1 + for arch in self.pkg_archs.split(): + config_file.write("arch %s %d\n" % (arch, priority)) + priority += 5 + + if self.d.has_key('IPK_FEED_URIS'): + ipk_feed_uris = self.d['IPK_FEED_URIS'] + else: + ipk_feed_uris = "" + + for line in ipk_feed_uris.split(): + feed_match = re.match("^[ \t]*(.*)##([^ \t]*)[ \t]*$", line) + + if feed_match is not None: + feed_name = feed_match.group(1) + feed_uri = feed_match.group(2) + + msger.info("Add %s feed with URL %s" % (feed_name, feed_uri)) + + config_file.write("src/gz %s %s\n" % (feed_name, feed_uri)) + + """ + Allow to use package deploy directory contents as quick devel-testing + feed. This creates individual feed configs for each arch subdir of those + specified as compatible for the current machine. + NOTE: Development-helper feature, NOT a full-fledged feed. + """ + if self.d.has_key('FEED_DEPLOYDIR_BASE_URI'): + feed_deploydir_base_dir = self.d['FEED_DEPLOYDIR_BASE_URI'] + else: + feed_deploydir_base_dir = "" + + if feed_deploydir_base_dir != "": + for arch in self.pkg_archs.split(): + if self.d.has_key("sysconfdir"): + sysconfdir = self.d["sysconfdir"] + else: + sysconfdir = None + + cfg_file_name = os.path.join(self.target_rootfs, + sysconfdir, + "opkg", + "local-%s-feed.conf" % arch) + + with open(cfg_file_name, "w+") as cfg_file: + cfg_file.write("src/gz local-%s %s/%s" % + arch, + feed_deploydir_base_dir, + arch) + + def _create_config(self): + with open(self.config_file, "w+") as config_file: + priority = 1 + for arch in self.pkg_archs.split(): + config_file.write("arch %s %d\n" % (arch, priority)) + priority += 5 + + config_file.write("src oe file:%s\n" % self.deploy_dir) + + for arch in self.pkg_archs.split(): + pkgs_dir = os.path.join(self.deploy_dir, arch) + if os.path.isdir(pkgs_dir): + config_file.write("src oe-%s file:%s\n" % + (arch, pkgs_dir)) + + def insert_feeds_uris(self): + if self.feed_uris == "": + return + + rootfs_config = os.path.join('%s/etc/opkg/base-feeds.conf' + % self.target_rootfs) + + with open(rootfs_config, "w+") as config_file: + uri_iterator = 0 + for uri in self.feed_uris.split(): + config_file.write("src/gz url-%d %s/ipk\n" % + (uri_iterator, uri)) + + for arch in self.pkg_archs.split(): + if not os.path.exists(os.path.join(self.deploy_dir, arch)): + continue + msger.info('Note: adding opkg channel url-%s-%d (%s)' % + (arch, uri_iterator, uri)) + + config_file.write("src/gz uri-%s-%d %s/ipk/%s\n" % + (arch, uri_iterator, uri, arch)) + uri_iterator += 1 + + def update(self): + self.deploy_dir_lock() + + cmd = "%s %s update" % (self.opkg_cmd, self.opkg_args) + + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) + if rc != 0: + self.deploy_dir_unlock() + msger.error("Unable to update the package index files. Command '%s' " + "returned %d" % (cmd, rc)) + + self.deploy_dir_unlock() + + def install(self, pkgs, attempt_only=False): + if attempt_only and len(pkgs) == 0: + return + + cmd = "%s %s install %s" % (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) + + os.environ['D'] = self.target_rootfs + os.environ['OFFLINE_ROOT'] = self.target_rootfs + os.environ['IPKG_OFFLINE_ROOT'] = self.target_rootfs + os.environ['OPKG_OFFLINE_ROOT'] = self.target_rootfs + if self.d.has_key('WORKDIR'): + os.environ['INTERCEPT_DIR'] = os.path.join(self.d['WORKDIR'], + "intercept_scripts") + else: + os.environ['INTERCEPT_DIR'] = "." + msger.warning("No WORKDIR provided!") + + if self.d.has_key('STAGING_DIR_NATIVE'): + os.environ['NATIVE_ROOT'] = self.d['STAGING_DIR_NATIVE'] + else: + msger.error("No STAGING_DIR_NATIVE provided!") + + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) + if rc != 0: + msger.error("Unable to install packages. " + "Command '%s' returned %d" % (cmd, rc)) + + + def remove(self, pkgs, with_dependencies=True): + if with_dependencies: + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \ + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) + else: + cmd = "%s %s --force-depends remove %s" % \ + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) + + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) + if rc != 0: + msger.error("Unable to remove packages. Command '%s' " + "returned %d" % (cmd, rc)) + + + def write_index(self): + self.deploy_dir_lock() + + result = self.indexer.write_index() + + self.deploy_dir_unlock() + + if result is not None: + msger.error(result) + + def remove_packaging_data(self): + remove(self.opkg_dir, True) + # create the directory back, it's needed by PM lock + mkdirhier(self.opkg_dir) + + def list_installed(self, format=None): + return WicOpkgPkgsList(self.d, self.target_rootfs, self.config_file).list(format) + + def handle_bad_recommendations(self): + if self.d.has_key("BAD_RECOMMENDATIONS"): + bad_recommendations = self.d["BAD_RECOMMENDATIONS"] + else: + bad_recommendations = "" + + if bad_recommendations.strip() == "": + return + + status_file = os.path.join(self.opkg_dir, "status") + + # If status file existed, it means the bad recommendations has already + # been handled + if os.path.exists(status_file): + return + + cmd = "%s %s info " % (self.opkg_cmd, self.opkg_args) + + with open(status_file, "w+") as status: + for pkg in bad_recommendations.split(): + pkg_info = cmd + pkg + + try: + output = subprocess.check_output(pkg_info.split(), stderr=subprocess.STDOUT).strip() + except subprocess.CalledProcessError as e: + msger.error("Cannot get package info. Command '%s' " + "returned %d:\n%s" % (pkg_info, e.returncode, e.output)) + + if output == "": + msger.info("Ignored bad recommendation: '%s' is " + "not a package" % pkg) + continue + + for line in output.split('\n'): + if line.startswith("Status:"): + status.write("Status: deinstall hold not-installed\n") + else: + status.write(line + "\n") + + ''' + The following function dummy installs pkgs and returns the log of output. + ''' + def dummy_install(self, pkgs): + if len(pkgs) == 0: + return + + # Create an temp dir as opkg root for dummy installation + if self.d.has_key("TMPDIR"): + tmp_dir = self.d["TMPDIR"] + else: + tmp_dir = "." + msger.warning("No TMPDIR provided!") + + temp_rootfs = '%s/opkg' % tmp_dir + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') + mkdirhier(temp_opkg_dir) + + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) + if self.d.has_key("OPKG_ARGS"): + opkg_args += self.d["OPKG_ARGS"] + + cmd = "%s %s update" % (self.opkg_cmd, opkg_args) + try: + subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) + except subprocess.CalledProcessError as e: + msger.error("Unable to update. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + # Dummy installation + cmd = "%s %s --noaction install %s " % (self.opkg_cmd, + opkg_args, + ' '.join(pkgs)) + try: + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) + except subprocess.CalledProcessError as e: + msger.error("Unable to dummy install packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + remove(temp_rootfs, True) + + return output + + def backup_packaging_data(self): + # Save the opkglib for increment ipk image generation + if os.path.exists(self.saved_opkg_dir): + remove(self.saved_opkg_dir, True) + shutil.copytree(self.opkg_dir, + self.saved_opkg_dir, + symlinks=True) + + def recover_packaging_data(self): + # Move the opkglib back + if os.path.exists(self.saved_opkg_dir): + if os.path.exists(self.opkg_dir): + remove(self.opkg_dir, True) + + msger.info('Recover packaging data') + shutil.copytree(self.saved_opkg_dir, + self.opkg_dir, + symlinks=True) + + +def wic_generate_index_files(d): + if d.has_key('PACKAGE_CLASSES'): + classes = d['PACKAGE_CLASSES'].replace("package_", "").split() + else: + classes = "" + msger.warning("No PACKAGE_CLASSES provided!") + + if d.has_key('DEPLOY_DIR_IPK'): + deploy_dir_ipk = d['DEPLOY_DIR_IPK'] + else: + deploy_dir_ipk = None + msger.warning("No DEPLOY_DIR_IPK provided!") + + indexer_map = { + "ipk": (WicOpkgIndexer, deploy_dir_ipk) + } + + result = None + + for pkg_class in classes: + if not pkg_class in indexer_map: + continue + + if os.path.exists(indexer_map[pkg_class][1]): + result = indexer_map[pkg_class][0](d, indexer_map[pkg_class][1]).write_index() + + if result is not None: + msger.error(result) diff --git a/scripts/wic b/scripts/wic index 2d3fd09..b9c8756 100755 --- a/scripts/wic +++ b/scripts/wic @@ -99,6 +99,10 @@ def wic_create_subcommand(args, usage_str): (options, args) = parser.parse_args(args) + if options.debug: + loglevel = logging.DEBUG + start_logging(loglevel) + if len(args) != 1: logging.error("Wrong number of arguments, exiting\n") parser.print_help() @@ -107,9 +111,11 @@ def wic_create_subcommand(args, usage_str): if not options.image_name and not (options.rootfs_dir and options.bootimg_dir and options.kernel_dir and + options.native_sysroot or options.native_sysroot): print "Build artifacts not completely specified, exiting." - print " (Use 'wic -e' or 'wic -r -b -k -n' to specify artifacts)" + print " (Use 'wic -e' or 'wic -r -b -k -n' or 'wic -r -n' to specify artifacts)" + print options sys.exit(1) if not options.image_name: @@ -125,13 +131,16 @@ def wic_create_subcommand(args, usage_str): print "Creating image(s)...\n" - bitbake_env_lines = find_bitbake_env_lines(options.image_name) - if not bitbake_env_lines: - print "Couldn't get bitbake environment, exiting." - sys.exit(1) - set_bitbake_env_lines(bitbake_env_lines) + # If '-e' option is used the values are extracted from bitbake env. + if options.image_name: + bitbake_env_lines = find_bitbake_env_lines(options.image_name) + if not bitbake_env_lines: + print "Couldn't get bitbake environment, exiting." + sys.exit(1) + set_bitbake_env_lines(bitbake_env_lines) bootimg_dir = staging_data_dir = hdddir = "" + rootfs_dir = native_sysroot = kernel_dir = image_output_dir = "" if options.image_name: (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot) = \ @@ -140,34 +149,65 @@ def wic_create_subcommand(args, usage_str): wks_file = args[0] if not wks_file.endswith(".wks"): + # Return full path of the .wks file wks_file = find_canned_image(scripts_path, wks_file) if not wks_file: - print "No image named %s found, exiting. (Use 'wic list images' to list available images, or specify a fully-qualified OE kickstart (.wks) filename)\n" % wks_file + print "No image named %s found, exiting.\n" % wks_file + print "(Use 'wic list images' to list available images, or specify a fully-qualified OE kickstart (.wks) filename)\n" sys.exit(1) - image_output_dir = "" if options.outdir: image_output_dir = options.outdir - if not options.image_name: - rootfs_dir = '' - if 'ROOTFS_DIR' in options.rootfs_dir: - rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] - bootimg_dir = options.bootimg_dir - kernel_dir = options.kernel_dir + if options.native_sysroot: native_sysroot = options.native_sysroot - if rootfs_dir and not os.path.isdir(rootfs_dir): - print "--roofs-dir (-r) not found, exiting\n" - sys.exit(1) - if not os.path.isdir(bootimg_dir): - print "--bootimg-dir (-b) not found, exiting\n" - sys.exit(1) - if not os.path.isdir(kernel_dir): - print "--kernel-dir (-k) not found, exiting\n" - sys.exit(1) + print "Using native_sysroot from user command: %s" % native_sysroot + if not os.path.isdir(native_sysroot): - print "--native-sysroot (-n) not found, exiting\n" + print "--native-sysroot (-n) not found, exiting" sys.exit(1) + + native_sysroot = os.path.abspath(native_sysroot) + + if not options.image_name: + if (options.bootimg_dir and options.kernel_dir and + options.rootfs_dir and options.native_sysroot): + rootfs_dir = '' + if 'ROOTFS_DIR' in options.rootfs_dir: + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] + bootimg_dir = options.bootimg_dir + kernel_dir = options.kernel_dir + native_sysroot = options.native_sysroot + + if rootfs_dir and not os.path.isdir(rootfs_dir): + print "--roofs-dir (-r) not found, exiting\n" + sys.exit(1) + if not os.path.isdir(bootimg_dir): + print "--bootimg-dir (-b) not found, exiting\n" + sys.exit(1) + if not os.path.isdir(kernel_dir): + print "--kernel-dir (-k) not found, exiting\n" + sys.exit(1) + if not os.path.isdir(native_sysroot): + print "--native-sysroot (-n) not found, exiting\n" + sys.exit(1) + else: + print 'Build image from rootfs and a package list using native rootfs\n' + if options.rootfs_dir and 'ROOTFS_DIR' in options.rootfs_dir: + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] + elif options.rootfs_dir: + rootfs_dir = options.rootfs_dir + else: + rootfs_dir = "" + + native_sysroot = options.native_sysroot + + if rootfs_dir and not os.path.isdir(rootfs_dir): + print "--roofs-dir (-r) not found, exiting\n" + sys.exit(1) + if not os.path.isdir(native_sysroot): + print "--native-sysroot (-n) not found, exiting\n" + sys.exit(1) else: not_found = not_found_dir = "" if not os.path.isdir(rootfs_dir): -- 1.7.10.4 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] Add wic support to generate rootfs image for uboot 2014-06-10 13:41 ` [PATCH 1/1] " Adrian Calianu @ 2014-06-12 2:49 ` Saul Wold 2014-06-12 8:43 ` [PATCHv2 0/2] wic: " y [not found] ` <53996893.88fd440a.0627.25d6SMTPIN_ADDED_BROKEN@mx.google.com> 0 siblings, 2 replies; 11+ messages in thread From: Saul Wold @ 2014-06-12 2:49 UTC (permalink / raw) To: Adrian Calianu, openembedded-core On 06/10/2014 06:41 AM, Adrian Calianu wrote: > Add support in wic to build ramdisk uboot images from list of packages or > from an existing rootfs folder. > > Some prerequisites are required for this new build setup: > ../poky/scripts/lib/image/config/wic.conf > [create] > arch=target_arch (example: powerpc, arm, x86) > pkgmgr=opkg > repourl=http://example.distro/p2020rdb/ipk/all http://example.distro/p2020rdb/ipk/p2020rdb http://example.distro/p2020rdb/ipk/ppce500v2 > > 1) Build an rootfs image from an existing bitbake build: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core-image-minimal -o .../output > > 2) Build an rootfs image from an existing rootfs and native_sysroot: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r .../path_to_rootfs -n .../path_to_native_sysroot -o .../output > > 3) Build an rootfs image only from a package list (on wks file): > > Add the package list to be installed as rootfs on ../poky/scripts/lib/image/canned-wks/uboot.wks: > %packages > packagegroup-core-boot > pramfs-init > run-postinsts > packagegroup-core-ssh-dropbear > %end > > Generate rootfs image: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n .../path_to_native_sysroot -o .../output > > Signed-off-by: Adrian Calianu <adrian.calianu@enea.com> > --- > scripts/lib/image/canned-wks/uboot.wks | 17 + > scripts/lib/image/config/wic.conf | 4 + > scripts/lib/image/engine.py | 7 +- > scripts/lib/mic/imager/direct.py | 4 + > .../lib/mic/kickstart/custom_commands/partition.py | 128 ++++ > scripts/lib/mic/plugin.py | 11 +- > scripts/lib/mic/pluginbase.py | 9 + > scripts/lib/mic/plugins/source/uboot.py | 173 +++++ > scripts/lib/mic/utils/oe/package_manager.py | 810 ++++++++++++++++++++ > scripts/wic | 88 ++- > 10 files changed, 1219 insertions(+), 32 deletions(-) > create mode 100644 scripts/lib/image/canned-wks/uboot.wks > create mode 100644 scripts/lib/mic/plugins/source/uboot.py > create mode 100644 scripts/lib/mic/utils/oe/package_manager.py > This patch does not seem to be applying cleanly, can you please verify that it was produced against the latest master? Thanks Sau! > diff --git a/scripts/lib/image/canned-wks/uboot.wks b/scripts/lib/image/canned-wks/uboot.wks > new file mode 100644 > index 0000000..7de0572 > --- /dev/null > +++ b/scripts/lib/image/canned-wks/uboot.wks > @@ -0,0 +1,17 @@ > +# short-description: . Create a ramdisk image for U-Boot > +# long-description: Creates a ramdisk image for U-Boot that user > +# can directly load it into ram through tftp > +# > +# part - is a wic command that drive the process of generating a valid file system > +# - --source=uboot : wic plugin that generates a ramdisk image for U-Boot > +# - --fstype=ext2 : file system type( ext2 / ext3 / ext 4) > +# > +# %packages %end - option to provide a list of packages that will be installed > +# into rootfs. All packages dependencies will be installed by > +# package manager(default opkg). > + > + > +part / --source=uboot --fstype=ext2 --label imageName --align 1024 > + > +%packages > +%end > diff --git a/scripts/lib/image/config/wic.conf b/scripts/lib/image/config/wic.conf > index e96d6ae..2a2750b 100644 > --- a/scripts/lib/image/config/wic.conf > +++ b/scripts/lib/image/config/wic.conf > @@ -4,4 +4,8 @@ distro_name = OpenEmbedded > > [create] > ; settings for create subcommand > +; repourl=http://linux.com/ipk/all http://linux.com/ipk/target http://linux.com/ipk/arch > +arch=powerpc > +pkgmgr=opkg > runtime=native > +install_pkgs=source > diff --git a/scripts/lib/image/engine.py b/scripts/lib/image/engine.py > index 3bda1bf..1256236 100644 > --- a/scripts/lib/image/engine.py > +++ b/scripts/lib/image/engine.py > @@ -96,9 +96,10 @@ def build_canned_image_list(dl): > layers_path = get_bitbake_var("BBLAYERS") > canned_wks_layer_dirs = [] > > - for layer_path in layers_path.split(): > - path = os.path.join(layer_path, SCRIPTS_CANNED_IMAGE_DIR) > - canned_wks_layer_dirs.append(path) > + if layers_path is not None: > + for layer_path in layers_path.split(): > + path = os.path.join(layer_path, SCRIPTS_CANNED_IMAGE_DIR) > + canned_wks_layer_dirs.append(path) > > path = os.path.join(dl, CANNED_IMAGE_DIR) > canned_wks_layer_dirs.append(path) > diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py > index 2cf4c8d..fef9d0e 100644 > --- a/scripts/lib/mic/imager/direct.py > +++ b/scripts/lib/mic/imager/direct.py > @@ -262,6 +262,10 @@ class DirectImageCreator(BaseImageCreator): > # when/if we need to actually do package selection we > # should modify things to use those objects, but for now > # we can avoid that. > + > + p.install_pkgs(self, self.workdir, self.oe_builddir, self.rootfs_dir, > + self.bootimg_dir, self.kernel_dir, self.native_sysroot) > + > p.prepare(self, self.workdir, self.oe_builddir, self.rootfs_dir, > self.bootimg_dir, self.kernel_dir, self.native_sysroot) > > diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py > index 6b575c0..450d2d4 100644 > --- a/scripts/lib/mic/kickstart/custom_commands/partition.py > +++ b/scripts/lib/mic/kickstart/custom_commands/partition.py > @@ -31,7 +31,11 @@ from mic.utils.oe.misc import * > from mic.kickstart.custom_commands import * > from mic.plugin import pluginmgr > > +import os > +from mic.utils.oe.package_manager import * > + > partition_methods = { > + "do_install_pkgs":None, > "do_stage_partition":None, > "do_prepare_partition":None, > "do_configure_partition":None, > @@ -115,6 +119,102 @@ class Wic_PartData(Mic_PartData): > else: > return 0 > > + def install_pkgs(self, creator, cr_workdir, oe_builddir, rootfs_dir, > + bootimg_dir, kernel_dir, native_sysroot): > + """ > + Prepare content for individual partitions, installing packages. > + """ > + > + if not self.source: > + return > + > + self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods) > + self._source_methods["do_install_pkgs"](self, creator, > + cr_workdir, > + oe_builddir, > + rootfs_dir, > + bootimg_dir, > + kernel_dir, > + native_sysroot) > + > + def install_pkgs_ipk(self, cr_workdir, oe_builddir, rootfs_dir, > + native_sysroot, packages, repourl): > + """ > + Install packages specified into wks file using opkg package manager. > + This method is dependend on bb module. > + """ > + > + gVar = {} > + gVar["DEPLOY_DIR_IPK"] = os.path.join(oe_builddir, "tmp/deploy/ipk") > + > + # Run postinstall scripts even in offline mode > + # Use the arch priority package rather than higher version one if more than one candidate is found. > + #d.setVar("OPKG_ARGS", "--force_postinstall --prefer-arch-to-version") > + gVar["OPKG_ARGS"] = "--force_postinstall" > + > + # OPKG path relative to /output_path > + gVar["OPKGLIBDIR"] = "var/lib" > + > + source_url = repourl.split() > + > + # Generate feed uri's names, it doesn't seem to matter what name they have > + feed_uris = "" > + cnt = 0 > + archs = "" > + for url in source_url: > + feed_uris += "cl_def_feed%d##%s\n" % (cnt, url) > + cnt += 1 > + head, tail = os.path.split(url) > + archs += " " + tail > + > + # IPK_FEED_URIS with special formating defines the URI's used as source for packages > + gVar['IPK_FEED_URIS'] = feed_uris > + > + gVar['BUILD_IMAGES_FROM_FEEDS'] = "1" > + > + # We need to provide sysroot for utilities > + gVar['STAGING_DIR_NATIVE'] = native_sysroot > + > + # Set WORKDIR for output > + gVar['WORKDIR'] = cr_workdir > + > + # Set TMPDIR for output > + gVar['TMPDIR'] = os.path.join(cr_workdir, "tmp") > + > + if 'ROOTFS_DIR' in rootfs_dir: > + target_dir = rootfs_dir['ROOTFS_DIR'] > + elif os.path.isdir(rootfs_dir): > + target_dir = rootfs_dir > + else: > + msg = "Couldn't find --rootfs-dir=%s connection" > + msg += " or it is not a valid path, exiting" > + msger.error(msg % rootfs_dir) > + > + # Need native sysroot /usr/bin/ for opkg-cl > + # chnage PATH var to avoid issues with host tools > + defpath = os.environ['PATH'] > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > + > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % target_dir > + pseudo += "export PSEUDO_PASSWD=%s;" % target_dir > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudo += "%s/usr/bin/pseudo " % native_sysroot > + > + pm = WicOpkgPM(gVar, > + target_dir, > + 'opkg.conf', > + archs, > + pseudo, > + native_sysroot) > + > + pm.update() > + > + pm.install(packages) > + > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > + > + > def prepare(self, cr, cr_workdir, oe_builddir, rootfs_dir, bootimg_dir, > kernel_dir, native_sysroot): > """ > @@ -225,6 +325,34 @@ class Wic_PartData(Mic_PartData): > > return 0 > > + def prepare_for_uboot(self, arch, cr_workdir, oe_builddir, rootfs_dir, > + native_sysroot): > + """ > + Generates u-boot image from source_file( ext2/3/4 ) > + > + """ > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir > + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudo += "%s/usr/bin/pseudo " % native_sysroot > + > + # 1) compress image > + rootfs = self.source_file > + rootfs_gzip = "%s.gz" % rootfs > + gzip_cmd = "gzip -f -9 -c %s > %s" % (rootfs, rootfs_gzip) > + rc, out = exec_native_cmd(pseudo + gzip_cmd, native_sysroot) > + > + # 2) image for U-Boot > + rootfs_uboot = "%s.u-boot" % rootfs_gzip > + mkimage_cmd = "mkimage -A %s -O linux -T ramdisk -C gzip -n %s -d %s %s" % \ > + (arch, self.label, rootfs_gzip, rootfs_uboot) > + rc, out = exec_native_cmd(pseudo + mkimage_cmd, native_sysroot) > + > + msger.info("\n\n\tThe new U-Boot ramdisk image can be found here:\n\t\t%s\n\n" % rootfs_uboot) > + > + return 0 > + > def prepare_rootfs_btrfs(self, cr_workdir, oe_builddir, rootfs_dir, > native_sysroot, pseudo): > """ > diff --git a/scripts/lib/mic/plugin.py b/scripts/lib/mic/plugin.py > index bec33d6..585fd6d 100644 > --- a/scripts/lib/mic/plugin.py > +++ b/scripts/lib/mic/plugin.py > @@ -53,12 +53,13 @@ class PluginMgr(object): > self.layers_path = get_bitbake_var("BBLAYERS") > layer_dirs = [] > > - for layer_path in self.layers_path.split(): > - path = os.path.join(layer_path, SCRIPTS_PLUGIN_DIR, ptype) > - layer_dirs.append(path) > + if self.layers_path is not None: > + for layer_path in self.layers_path.split(): > + path = os.path.join(layer_path, SCRIPTS_PLUGIN_DIR, ptype) > + layer_dirs.append(path) > > - path = os.path.join(dl, ptype) > - layer_dirs.append(path) > + path = os.path.join(dl, ptype) > + layer_dirs.append(path) > > return layer_dirs > > diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py > index 9cf4c62..881d996 100644 > --- a/scripts/lib/mic/pluginbase.py > +++ b/scripts/lib/mic/pluginbase.py > @@ -89,6 +89,15 @@ class SourcePlugin(_Plugin): > """ > > @classmethod > + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, rootfs_dir, > + bootimg_dir, kernel_dir, native_sysroot): > + """ > + Called before partitions have been prepared and assembled into a > + disk image. Install packages into rootfs > + """ > + msger.debug("SourcePlugin: do_install_pkgs: part %s" % part) > + > + @classmethod > def do_install_disk(self, disk, disk_name, cr, workdir, oe_builddir, > bootimg_dir, kernel_dir, native_sysroot): > """ > diff --git a/scripts/lib/mic/plugins/source/uboot.py b/scripts/lib/mic/plugins/source/uboot.py > new file mode 100644 > index 0000000..57cb3cf > --- /dev/null > +++ b/scripts/lib/mic/plugins/source/uboot.py > @@ -0,0 +1,173 @@ > +# ex:ts=4:sw=4:sts=4:et > +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > +# > +# Copyright (c) 2014, Enea AB. > +# All rights reserved. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License version 2 as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > +# > +# DESCRIPTION > +# This implements the 'uboot' source plugin class for 'wic' > +# > +# AUTHORS > +# Adrian Calianu <adrian.calianu (at] enea.com> > +# > + > +import os > +import shutil > +import re > +import tempfile > + > +from mic import kickstart, chroot, msger > +from mic.utils import misc, fs_related, errors, runner, cmdln > +from mic.conf import configmgr > +from mic.plugin import pluginmgr > +from mic.utils.partitionedfs import PartitionedMount > +import mic.imager.direct as direct > +from mic.pluginbase import SourcePlugin > +from mic.utils.oe.misc import * > +from mic.imager.direct import DirectImageCreator > + > +def create_local_rootfs(part, creator, cr_workdir, krootfs_dir, native_sysroot): > + # In order to have a full control over rootfs we will make a local copy under workdir > + # and change rootfs_dir to new location. > + # In this way we can install more than one ROOTFS_DIRs and/or use > + # an empty rootfs to install packages, so a rootfs could be generated only from pkgs > + # TBD: create workdir/rootfs ; copy rootfs-> workdir/rootfs; set rootfs=workdir/rootfs > + > + cr_workdir = os.path.abspath(cr_workdir) > + new_rootfs_dir = "%s/rootfs_%s" % (cr_workdir, creator.name) > + > + rootfs_exists = 1 > + if part.rootfs is None: > + if not 'ROOTFS_DIR' in krootfs_dir: > + msg = "Couldn't find --rootfs-dir, exiting, " > + msger.info(msg) > + rootfs_exists = 0 > + rootfs_dir = krootfs_dir['ROOTFS_DIR'] > + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir > + else: > + if part.rootfs in krootfs_dir: > + rootfs_dir = krootfs_dir[part.rootfs] > + creator.rootfs_dir[part.rootfs] = new_rootfs_dir > + elif os.path.isdir(part.rootfs): > + rootfs_dir = part.rootfs > + part.rootfs = new_rootfs_dir > + else: > + msg = "Couldn't find --rootfs-dir=%s connection" > + msg += " or it is not a valid path, exiting" > + msger.info(msg % part.rootfs) > + rootfs_exists = 0 > + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir > + > + pseudox = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudox += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % new_rootfs_dir > + pseudox += "export PSEUDO_PASSWD=%s;" % new_rootfs_dir > + pseudox += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudox += "%s/usr/bin/pseudo " % native_sysroot > + > + mkdir_cmd = "mkdir %s" % (new_rootfs_dir) > + # rc, out = exec_native_cmd(pseudox + mkdir_cmd, native_sysroot) > + rc, out = exec_cmd(mkdir_cmd, True) > + > + if rootfs_exists == 1 and os.path.isdir(rootfs_dir): > + defpath = os.environ['PATH'] > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > + > + rootfs_dir = os.path.abspath(rootfs_dir) > + > + pseudoc = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudoc += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir > + pseudoc += "export PSEUDO_PASSWD=%s;" % rootfs_dir > + pseudoc += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudoc += "%s/usr/bin/pseudo " % native_sysroot > + > + tarc_cmd = "tar cvpf %s/rootfs.tar -C %s ." % (cr_workdir, rootfs_dir) > + rc, out = exec_native_cmd(pseudoc + tarc_cmd, native_sysroot) > + > + tarx_cmd = "tar xpvf %s/rootfs.tar -C %s" % (cr_workdir, new_rootfs_dir) > + rc, out = exec_native_cmd(pseudox + tarx_cmd, native_sysroot) > + > + rm_cmd = "rm %s/rootfs.tar" % cr_workdir > + rc, out = exec_cmd(rm_cmd, True) > + > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > + > + return new_rootfs_dir > + > +class UBootPlugin(SourcePlugin): > + name = 'uboot' > + > + @classmethod > + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, krootfs_dir, > + bootimg_dir, kernel_dir, native_sysroot): > + """ > + Called before all partitions have been prepared and assembled into a > + disk image. Intall packages based on wic configuration. > + """ > + > + # set new rootfs_dir > + rootfs_dir = create_local_rootfs(part, creator, cr_workdir, krootfs_dir, native_sysroot) > + > + # wks file parsing > + packages = kickstart.get_packages(creator.ks) > + > + # wic.conf file parsing = found under 'creator' > + local_pkgs_path = creator._local_pkgs_path > + repourl = creator.repourl > + pkgmgr = creator.pkgmgr_name > + > + # install packages > + if packages and pkgmgr in ["opkg"]: > + if len(repourl) > 0 : > + part.install_pkgs_ipk(cr_workdir, oe_builddir, rootfs_dir, native_sysroot, > + packages, repourl) > + else: > + msger.error("No packages repository provided in wic.conf") > + > + @classmethod > + def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir, > + kernel_dir, krootfs_dir, native_sysroot): > + """ > + Called to do the actual content population for a partition i.e. it > + 'prepares' the partition to be incorporated into the image. > + In this case, prepare content for legacy bios boot partition. > + """ > + if part.rootfs is None: > + if not 'ROOTFS_DIR' in krootfs_dir: > + msg = "Couldn't find --rootfs-dir, exiting" > + msger.error(msg) > + rootfs_dir = krootfs_dir['ROOTFS_DIR'] > + else: > + if part.rootfs in krootfs_dir: > + rootfs_dir = krootfs_dir[part.rootfs] > + elif os.path.isdir(part.rootfs): > + rootfs_dir = part.rootfs > + else: > + msg = "Couldn't find --rootfs-dir=%s connection" > + msg += " or it is not a valid path, exiting" > + msger.error(msg % part.rootfs) > + > + part.set_rootfs(rootfs_dir) > + > + # change partition label wich will reflect into the final rootfs image name > + part.label = "%s_%s" % (part.label, cr.name) > + > + defpath = os.environ['PATH'] > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > + > + part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot) > + part.prepare_for_uboot(cr.target_arch,cr_workdir, oe_builddir, rootfs_dir, native_sysroot) > + > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > diff --git a/scripts/lib/mic/utils/oe/package_manager.py b/scripts/lib/mic/utils/oe/package_manager.py > new file mode 100644 > index 0000000..92ce98e > --- /dev/null > +++ b/scripts/lib/mic/utils/oe/package_manager.py > @@ -0,0 +1,810 @@ > +# ex:ts=4:sw=4:sts=4:et > +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > +# > +# Copyright (c) 2014, Enea AB. > +# All rights reserved. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License version 2 as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > +# > +# DESCRIPTION > +# This implements the opkg package manager wrapper as a combination of > +# meta/lib/oe/package_manager.py and bitbake/lib/bb/utils.py files and > +# adaptation of those files to 'wic'. > +# > +# AUTHORS > +# Adrian Calianu <adrian.calianu (at] enea.com> > +# > +# This file incorporates work covered by the following copyright and > +# permission notice: > +# > +# meta/COPYING.GPLv2 (GPLv2) > +# meta/COPYING.MIT (MIT) > +# > +# Copyright (C) 2004 Michael Lauer > +# > +# Permission to use, copy, modify, and/or distribute this software > +# for any purpose with or without fee is hereby granted, provided > +# that the above copyright notice and this permission notice appear > +# in all copies. > +# > +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL > +# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED > +# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE > +# AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR > +# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS > +# OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, > +# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN > +# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. > + > + > +from abc import ABCMeta, abstractmethod > +import os > +import glob > +import subprocess > +import shutil > +import multiprocessing > +import re > +import errno > +import fcntl > + > +from mic.utils.oe.misc import * > +from mic import msger > + > +def mkdirhier(directory): > + """Create a directory like 'mkdir -p', but does not complain if > + directory already exists like os.makedirs > + """ > + > + try: > + os.makedirs(directory) > + except OSError as e: > + if e.errno != errno.EEXIST: > + raise e > + > +def remove(path, recurse=False): > + """Equivalent to rm -f or rm -rf""" > + if not path: > + return > + if recurse: > + # shutil.rmtree(name) would be ideal but its too slow > + subprocess.call(['rm', '-rf'] + glob.glob(path)) > + return > + for name in glob.glob(path): > + try: > + os.unlink(name) > + except OSError as exc: > + if exc.errno != errno.ENOENT: > + raise > + > +def lockfile(name, shared=False, retry=True): > + """ > + Use the file fn as a lock file, return when the lock has been acquired. > + Returns a variable to pass to unlockfile(). > + """ > + dirname = os.path.dirname(name) > + mkdirhier(dirname) > + > + if not os.access(dirname, os.W_OK): > + logger.error("Unable to acquire lock '%s', directory is not writable", > + name) > + sys.exit(1) > + > + op = fcntl.LOCK_EX > + if shared: > + op = fcntl.LOCK_SH > + if not retry: > + op = op | fcntl.LOCK_NB > + > + while True: > + # If we leave the lockfiles lying around there is no problem > + # but we should clean up after ourselves. This gives potential > + # for races though. To work around this, when we acquire the lock > + # we check the file we locked was still the lock file on disk. > + # by comparing inode numbers. If they don't match or the lockfile > + # no longer exists, we start again. > + > + # This implementation is unfair since the last person to request the > + # lock is the most likely to win it. > + > + try: > + lf = open(name, 'a+') > + fileno = lf.fileno() > + fcntl.flock(fileno, op) > + statinfo = os.fstat(fileno) > + if os.path.exists(lf.name): > + statinfo2 = os.stat(lf.name) > + if statinfo.st_ino == statinfo2.st_ino: > + return lf > + lf.close() > + except Exception: > + try: > + lf.close() > + except Exception: > + pass > + pass > + if not retry: > + return None > + > +def unlockfile(lf): > + """ > + Unlock a file locked using lockfile() > + """ > + try: > + # If we had a shared lock, we need to promote to exclusive before > + # removing the lockfile. Attempt this, ignore failures. > + fcntl.flock(lf.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB) > + os.unlink(lf.name) > + except (IOError, OSError): > + pass > + fcntl.flock(lf.fileno(), fcntl.LOCK_UN) > + lf.close() > + > +def which(path, item, direction = 0, history = False): > + """ > + Locate a file in a PATH > + """ > + > + hist = [] > + paths = (path or "").split(':') > + if direction != 0: > + paths.reverse() > + > + for p in paths: > + next = os.path.join(p, item) > + hist.append(next) > + if os.path.exists(next): > + if not os.path.isabs(next): > + next = os.path.abspath(next) > + if history: > + return next, hist > + return next > + > + if history: > + return "", hist > + return "" > + > + > + > +# this can be used by all PM backends to create the index files in parallel > +def wic_create_index(arg): > + index_cmd = arg > + > + try: > + msger.info("Executing '%s' ..." % index_cmd) > + subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True) > + except subprocess.CalledProcessError as e: > + return("Index creation command '%s' failed with return code %d:\n%s" % > + (e.cmd, e.returncode, e.output)) > + > + return None > + > + > +class WicIndexer(object): > + __metaclass__ = ABCMeta > + > + def __init__(self, d, deploy_dir): > + self.d = d > + self.deploy_dir = deploy_dir > + > + @abstractmethod > + def write_index(self): > + pass > + > +class WicOpkgIndexer(WicIndexer): > + def write_index(self): > + arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS", > + "SDK_PACKAGE_ARCHS", > + "MULTILIB_ARCHS"] > + > + opkg_index_cmd = which(os.getenv('PATH'), "opkg-make-index") > + > + if not os.path.exists(os.path.join(self.deploy_dir, "Packages")): > + open(os.path.join(self.deploy_dir, "Packages"), "w").close() > + > + index_cmds = [] > + for arch_var in arch_vars: > + if self.d.has_key(arch_var): > + archs = self.d[arch_var] > + else: > + archs = None > + > + if archs is None: > + continue > + > + for arch in archs.split(): > + pkgs_dir = os.path.join(self.deploy_dir, arch) > + pkgs_file = os.path.join(pkgs_dir, "Packages") > + > + if not os.path.isdir(pkgs_dir): > + continue > + > + if not os.path.exists(pkgs_file): > + open(pkgs_file, "w").close() > + > + index_cmds.append('%s -r %s -p %s -m %s' % > + (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir)) > + > + if len(index_cmds) == 0: > + msger.info("There are no packages in %s!" % self.deploy_dir) > + return > + > + nproc = multiprocessing.cpu_count() > + pool = multiprocessing.Pool(nproc) > + results = list(pool.imap(wic_create_index, index_cmds)) > + pool.close() > + pool.join() > + > + for result in results: > + if result is not None: > + return(result) > + > +class WicPkgsList(object): > + __metaclass__ = ABCMeta > + > + def __init__(self, d, rootfs_dir): > + self.d = d > + self.rootfs_dir = rootfs_dir > + > + @abstractmethod > + def list(self, format=None): > + pass > + > + > +class WicOpkgPkgsList(WicPkgsList): > + def __init__(self, d, rootfs_dir, config_file): > + super(WicOpkgPkgsList, self).__init__(d, rootfs_dir) > + > + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") > + self.opkg_args = "-f %s -o %s " % (config_file, rootfs_dir) > + if self.d.has_key("OPKG_ARGS"): > + self.opkg_args += self.d["OPKG_ARGS"] > + > + def list(self, format=None): > + opkg_query_cmd = which(os.getenv('PATH'), "opkg-query-helper.py") > + > + if format == "arch": > + cmd = "%s %s status | %s -a" % \ > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > + elif format == "file": > + cmd = "%s %s status | %s -f" % \ > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > + elif format == "ver": > + cmd = "%s %s status | %s -v" % \ > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > + elif format == "deps": > + cmd = "%s %s status | %s" % \ > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > + else: > + cmd = "%s %s list_installed | cut -d' ' -f1" % \ > + (self.opkg_cmd, self.opkg_args) > + > + try: > + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip() > + except subprocess.CalledProcessError as e: > + msger.error("Cannot get the installed packages list. Command '%s' " > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > + > + if output and format == "file": > + tmp_output = "" > + for line in output.split('\n'): > + pkg, pkg_file, pkg_arch = line.split() > + full_path = os.path.join(self.rootfs_dir, pkg_arch, pkg_file) > + if os.path.exists(full_path): > + tmp_output += "%s %s %s\n" % (pkg, full_path, pkg_arch) > + else: > + tmp_output += "%s %s %s\n" % (pkg, pkg_file, pkg_arch) > + > + output = tmp_output > + > + return output > + > + > +class WicPackageManager(object): > + """ > + This is an abstract class. Do not instantiate this directly. > + """ > + __metaclass__ = ABCMeta > + > + def __init__(self, d, pseudo, native_sysroot): > + self.d = d > + self.deploy_dir = None > + self.deploy_lock = None > + if self.d.has_key('PACKAGE_FEED_URIS'): > + self.feed_uris = self.d['PACKAGE_FEED_URIS'] > + else: > + self.feed_uris = "" > + self.pseudo = pseudo > + self.native_sysroot = native_sysroot > + > + """ > + Update the package manager package database. > + """ > + @abstractmethod > + def update(self): > + pass > + > + """ > + Install a list of packages. 'pkgs' is a list object. If 'attempt_only' is > + True, installation failures are ignored. > + """ > + @abstractmethod > + def install(self, pkgs, attempt_only=False): > + pass > + > + """ > + Remove a list of packages. 'pkgs' is a list object. If 'with_dependencies' > + is False, the any dependencies are left in place. > + """ > + @abstractmethod > + def remove(self, pkgs, with_dependencies=True): > + pass > + > + """ > + This function creates the index files > + """ > + @abstractmethod > + def write_index(self): > + pass > + > + @abstractmethod > + def remove_packaging_data(self): > + pass > + > + @abstractmethod > + def list_installed(self, format=None): > + pass > + > + @abstractmethod > + def insert_feeds_uris(self): > + pass > + > + """ > + Install complementary packages based upon the list of currently installed > + packages e.g. locales, *-dev, *-dbg, etc. This will only attempt to install > + these packages, if they don't exist then no error will occur. Note: every > + backend needs to call this function explicitly after the normal package > + installation > + """ > + def install_complementary(self, globs=None): > + # we need to write the list of installed packages to a file because the > + # oe-pkgdata-util reads it from a file > + if self.d.has_key('WORKDIR'): > + installed_pkgs_file = os.path.join(self.d['WORKDIR'], > + "installed_pkgs.txt") > + else: > + msger.error("No WORKDIR provided!") > + > + with open(installed_pkgs_file, "w+") as installed_pkgs: > + installed_pkgs.write(self.list_installed("arch")) > + > + if globs is None: > + if self.d.has_key('IMAGE_INSTALL_COMPLEMENTARY'): > + globs = self.d['IMAGE_INSTALL_COMPLEMENTARY'] > + split_linguas = set() > + > + if self.d.has_key('IMAGE_LINGUAS'): > + for translation in self.d['IMAGE_LINGUAS'].split(): > + split_linguas.add(translation) > + split_linguas.add(translation.split('-')[0]) > + > + split_linguas = sorted(split_linguas) > + > + for lang in split_linguas: > + globs += " *-locale-%s" % lang > + > + if globs is None: > + return > + > + if not self.d.has_key('PKGDATA_DIR'): > + msger.error("No PKGDATA_DIR provided!") > + > + cmd = [which(os.getenv('PATH'), "oe-pkgdata-util"), > + "glob", self.d['PKGDATA_DIR'], installed_pkgs_file, > + globs] > + > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > + if rc != 0: > + msger.error("Could not compute complementary packages list. Command " > + "'%s' returned %d" % > + (' '.join(cmd), rc)) > + > + self.install(out.split(), attempt_only=True) > + > + > + def deploy_dir_lock(self): > + if self.deploy_dir is None: > + raise RuntimeError("deploy_dir is not set!") > + > + lock_file_name = os.path.join(self.deploy_dir, "deploy.lock") > + > + self.deploy_lock = lockfile(lock_file_name) > + > + def deploy_dir_unlock(self): > + if self.deploy_lock is None: > + return > + > + unlockfile(self.deploy_lock) > + > + self.deploy_lock = None > + > + > +class WicOpkgPM(WicPackageManager): > + def __init__(self, d, target_rootfs, config_file, archs, pseudo, native_sysroot, task_name='target'): > + super(WicOpkgPM, self).__init__(d, pseudo, native_sysroot) > + > + self.target_rootfs = target_rootfs > + self.config_file = config_file > + self.pkg_archs = archs > + self.task_name = task_name > + > + if self.d.has_key("DEPLOY_DIR_IPK"): > + self.deploy_dir = self.d["DEPLOY_DIR_IPK"] > + > + self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock") > + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") > + self.opkg_args = "-f %s -o %s " % (self.config_file, target_rootfs) > + if self.d.has_key("OPKG_ARGS"): > + self.opkg_args += self.d["OPKG_ARGS"] > + > + if self.d.has_key('OPKGLIBDIR'): > + opkg_lib_dir = self.d['OPKGLIBDIR'] > + else: > + opkg_lib_dir = "" > + > + if opkg_lib_dir[0] == "/": > + opkg_lib_dir = opkg_lib_dir[1:] > + > + self.opkg_dir = os.path.join(target_rootfs, opkg_lib_dir, "opkg") > + > + mkdirhier(self.opkg_dir) > + > + if self.d.has_key("TMPDIR"): > + tmp_dir = self.d["TMPDIR"] > + else: > + tmp_dir = "" > + > + self.saved_opkg_dir = '%s/saved/%s' % (tmp_dir, self.task_name) > + if not os.path.exists('%s/saved' % tmp_dir): > + mkdirhier('%s/saved' % tmp_dir) > + > + if self.d.has_key('BUILD_IMAGES_FROM_FEEDS') and self.d['BUILD_IMAGES_FROM_FEEDS'] != "1": > + self._create_config() > + else: > + self._create_custom_config() > + > + self.indexer = WicOpkgIndexer(self.d, self.deploy_dir) > + > + """ > + This function will change a package's status in /var/lib/opkg/status file. > + If 'packages' is None then the new_status will be applied to all > + packages > + """ > + def mark_packages(self, status_tag, packages=None): > + status_file = os.path.join(self.opkg_dir, "status") > + > + with open(status_file, "r") as sf: > + with open(status_file + ".tmp", "w+") as tmp_sf: > + if packages is None: > + tmp_sf.write(re.sub(r"Package: (.*?)\n((?:[^\n]+\n)*?)Status: (.*)(?:unpacked|installed)", > + r"Package: \1\n\2Status: \3%s" % status_tag, > + sf.read())) > + else: > + if type(packages).__name__ != "list": > + raise TypeError("'packages' should be a list object") > + > + status = sf.read() > + for pkg in packages: > + status = re.sub(r"Package: %s\n((?:[^\n]+\n)*?)Status: (.*)(?:unpacked|installed)" % pkg, > + r"Package: %s\n\1Status: \2%s" % (pkg, status_tag), > + status) > + > + tmp_sf.write(status) > + > + os.rename(status_file + ".tmp", status_file) > + > + def _create_custom_config(self): > + msger.info("Building from feeds activated!") > + > + with open(self.config_file, "w+") as config_file: > + priority = 1 > + for arch in self.pkg_archs.split(): > + config_file.write("arch %s %d\n" % (arch, priority)) > + priority += 5 > + > + if self.d.has_key('IPK_FEED_URIS'): > + ipk_feed_uris = self.d['IPK_FEED_URIS'] > + else: > + ipk_feed_uris = "" > + > + for line in ipk_feed_uris.split(): > + feed_match = re.match("^[ \t]*(.*)##([^ \t]*)[ \t]*$", line) > + > + if feed_match is not None: > + feed_name = feed_match.group(1) > + feed_uri = feed_match.group(2) > + > + msger.info("Add %s feed with URL %s" % (feed_name, feed_uri)) > + > + config_file.write("src/gz %s %s\n" % (feed_name, feed_uri)) > + > + """ > + Allow to use package deploy directory contents as quick devel-testing > + feed. This creates individual feed configs for each arch subdir of those > + specified as compatible for the current machine. > + NOTE: Development-helper feature, NOT a full-fledged feed. > + """ > + if self.d.has_key('FEED_DEPLOYDIR_BASE_URI'): > + feed_deploydir_base_dir = self.d['FEED_DEPLOYDIR_BASE_URI'] > + else: > + feed_deploydir_base_dir = "" > + > + if feed_deploydir_base_dir != "": > + for arch in self.pkg_archs.split(): > + if self.d.has_key("sysconfdir"): > + sysconfdir = self.d["sysconfdir"] > + else: > + sysconfdir = None > + > + cfg_file_name = os.path.join(self.target_rootfs, > + sysconfdir, > + "opkg", > + "local-%s-feed.conf" % arch) > + > + with open(cfg_file_name, "w+") as cfg_file: > + cfg_file.write("src/gz local-%s %s/%s" % > + arch, > + feed_deploydir_base_dir, > + arch) > + > + def _create_config(self): > + with open(self.config_file, "w+") as config_file: > + priority = 1 > + for arch in self.pkg_archs.split(): > + config_file.write("arch %s %d\n" % (arch, priority)) > + priority += 5 > + > + config_file.write("src oe file:%s\n" % self.deploy_dir) > + > + for arch in self.pkg_archs.split(): > + pkgs_dir = os.path.join(self.deploy_dir, arch) > + if os.path.isdir(pkgs_dir): > + config_file.write("src oe-%s file:%s\n" % > + (arch, pkgs_dir)) > + > + def insert_feeds_uris(self): > + if self.feed_uris == "": > + return > + > + rootfs_config = os.path.join('%s/etc/opkg/base-feeds.conf' > + % self.target_rootfs) > + > + with open(rootfs_config, "w+") as config_file: > + uri_iterator = 0 > + for uri in self.feed_uris.split(): > + config_file.write("src/gz url-%d %s/ipk\n" % > + (uri_iterator, uri)) > + > + for arch in self.pkg_archs.split(): > + if not os.path.exists(os.path.join(self.deploy_dir, arch)): > + continue > + msger.info('Note: adding opkg channel url-%s-%d (%s)' % > + (arch, uri_iterator, uri)) > + > + config_file.write("src/gz uri-%s-%d %s/ipk/%s\n" % > + (arch, uri_iterator, uri, arch)) > + uri_iterator += 1 > + > + def update(self): > + self.deploy_dir_lock() > + > + cmd = "%s %s update" % (self.opkg_cmd, self.opkg_args) > + > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > + if rc != 0: > + self.deploy_dir_unlock() > + msger.error("Unable to update the package index files. Command '%s' " > + "returned %d" % (cmd, rc)) > + > + self.deploy_dir_unlock() > + > + def install(self, pkgs, attempt_only=False): > + if attempt_only and len(pkgs) == 0: > + return > + > + cmd = "%s %s install %s" % (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) > + > + os.environ['D'] = self.target_rootfs > + os.environ['OFFLINE_ROOT'] = self.target_rootfs > + os.environ['IPKG_OFFLINE_ROOT'] = self.target_rootfs > + os.environ['OPKG_OFFLINE_ROOT'] = self.target_rootfs > + if self.d.has_key('WORKDIR'): > + os.environ['INTERCEPT_DIR'] = os.path.join(self.d['WORKDIR'], > + "intercept_scripts") > + else: > + os.environ['INTERCEPT_DIR'] = "." > + msger.warning("No WORKDIR provided!") > + > + if self.d.has_key('STAGING_DIR_NATIVE'): > + os.environ['NATIVE_ROOT'] = self.d['STAGING_DIR_NATIVE'] > + else: > + msger.error("No STAGING_DIR_NATIVE provided!") > + > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > + if rc != 0: > + msger.error("Unable to install packages. " > + "Command '%s' returned %d" % (cmd, rc)) > + > + > + def remove(self, pkgs, with_dependencies=True): > + if with_dependencies: > + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \ > + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) > + else: > + cmd = "%s %s --force-depends remove %s" % \ > + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) > + > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > + if rc != 0: > + msger.error("Unable to remove packages. Command '%s' " > + "returned %d" % (cmd, rc)) > + > + > + def write_index(self): > + self.deploy_dir_lock() > + > + result = self.indexer.write_index() > + > + self.deploy_dir_unlock() > + > + if result is not None: > + msger.error(result) > + > + def remove_packaging_data(self): > + remove(self.opkg_dir, True) > + # create the directory back, it's needed by PM lock > + mkdirhier(self.opkg_dir) > + > + def list_installed(self, format=None): > + return WicOpkgPkgsList(self.d, self.target_rootfs, self.config_file).list(format) > + > + def handle_bad_recommendations(self): > + if self.d.has_key("BAD_RECOMMENDATIONS"): > + bad_recommendations = self.d["BAD_RECOMMENDATIONS"] > + else: > + bad_recommendations = "" > + > + if bad_recommendations.strip() == "": > + return > + > + status_file = os.path.join(self.opkg_dir, "status") > + > + # If status file existed, it means the bad recommendations has already > + # been handled > + if os.path.exists(status_file): > + return > + > + cmd = "%s %s info " % (self.opkg_cmd, self.opkg_args) > + > + with open(status_file, "w+") as status: > + for pkg in bad_recommendations.split(): > + pkg_info = cmd + pkg > + > + try: > + output = subprocess.check_output(pkg_info.split(), stderr=subprocess.STDOUT).strip() > + except subprocess.CalledProcessError as e: > + msger.error("Cannot get package info. Command '%s' " > + "returned %d:\n%s" % (pkg_info, e.returncode, e.output)) > + > + if output == "": > + msger.info("Ignored bad recommendation: '%s' is " > + "not a package" % pkg) > + continue > + > + for line in output.split('\n'): > + if line.startswith("Status:"): > + status.write("Status: deinstall hold not-installed\n") > + else: > + status.write(line + "\n") > + > + ''' > + The following function dummy installs pkgs and returns the log of output. > + ''' > + def dummy_install(self, pkgs): > + if len(pkgs) == 0: > + return > + > + # Create an temp dir as opkg root for dummy installation > + if self.d.has_key("TMPDIR"): > + tmp_dir = self.d["TMPDIR"] > + else: > + tmp_dir = "." > + msger.warning("No TMPDIR provided!") > + > + temp_rootfs = '%s/opkg' % tmp_dir > + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') > + mkdirhier(temp_opkg_dir) > + > + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) > + if self.d.has_key("OPKG_ARGS"): > + opkg_args += self.d["OPKG_ARGS"] > + > + cmd = "%s %s update" % (self.opkg_cmd, opkg_args) > + try: > + subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) > + except subprocess.CalledProcessError as e: > + msger.error("Unable to update. Command '%s' " > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > + > + # Dummy installation > + cmd = "%s %s --noaction install %s " % (self.opkg_cmd, > + opkg_args, > + ' '.join(pkgs)) > + try: > + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) > + except subprocess.CalledProcessError as e: > + msger.error("Unable to dummy install packages. Command '%s' " > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > + > + remove(temp_rootfs, True) > + > + return output > + > + def backup_packaging_data(self): > + # Save the opkglib for increment ipk image generation > + if os.path.exists(self.saved_opkg_dir): > + remove(self.saved_opkg_dir, True) > + shutil.copytree(self.opkg_dir, > + self.saved_opkg_dir, > + symlinks=True) > + > + def recover_packaging_data(self): > + # Move the opkglib back > + if os.path.exists(self.saved_opkg_dir): > + if os.path.exists(self.opkg_dir): > + remove(self.opkg_dir, True) > + > + msger.info('Recover packaging data') > + shutil.copytree(self.saved_opkg_dir, > + self.opkg_dir, > + symlinks=True) > + > + > +def wic_generate_index_files(d): > + if d.has_key('PACKAGE_CLASSES'): > + classes = d['PACKAGE_CLASSES'].replace("package_", "").split() > + else: > + classes = "" > + msger.warning("No PACKAGE_CLASSES provided!") > + > + if d.has_key('DEPLOY_DIR_IPK'): > + deploy_dir_ipk = d['DEPLOY_DIR_IPK'] > + else: > + deploy_dir_ipk = None > + msger.warning("No DEPLOY_DIR_IPK provided!") > + > + indexer_map = { > + "ipk": (WicOpkgIndexer, deploy_dir_ipk) > + } > + > + result = None > + > + for pkg_class in classes: > + if not pkg_class in indexer_map: > + continue > + > + if os.path.exists(indexer_map[pkg_class][1]): > + result = indexer_map[pkg_class][0](d, indexer_map[pkg_class][1]).write_index() > + > + if result is not None: > + msger.error(result) > diff --git a/scripts/wic b/scripts/wic > index 2d3fd09..b9c8756 100755 > --- a/scripts/wic > +++ b/scripts/wic > @@ -99,6 +99,10 @@ def wic_create_subcommand(args, usage_str): > > (options, args) = parser.parse_args(args) > > + if options.debug: > + loglevel = logging.DEBUG > + start_logging(loglevel) > + > if len(args) != 1: > logging.error("Wrong number of arguments, exiting\n") > parser.print_help() > @@ -107,9 +111,11 @@ def wic_create_subcommand(args, usage_str): > if not options.image_name and not (options.rootfs_dir and > options.bootimg_dir and > options.kernel_dir and > + options.native_sysroot or > options.native_sysroot): > print "Build artifacts not completely specified, exiting." > - print " (Use 'wic -e' or 'wic -r -b -k -n' to specify artifacts)" > + print " (Use 'wic -e' or 'wic -r -b -k -n' or 'wic -r -n' to specify artifacts)" > + print options > sys.exit(1) > > if not options.image_name: > @@ -125,13 +131,16 @@ def wic_create_subcommand(args, usage_str): > > print "Creating image(s)...\n" > > - bitbake_env_lines = find_bitbake_env_lines(options.image_name) > - if not bitbake_env_lines: > - print "Couldn't get bitbake environment, exiting." > - sys.exit(1) > - set_bitbake_env_lines(bitbake_env_lines) > + # If '-e' option is used the values are extracted from bitbake env. > + if options.image_name: > + bitbake_env_lines = find_bitbake_env_lines(options.image_name) > + if not bitbake_env_lines: > + print "Couldn't get bitbake environment, exiting." > + sys.exit(1) > + set_bitbake_env_lines(bitbake_env_lines) > > bootimg_dir = staging_data_dir = hdddir = "" > + rootfs_dir = native_sysroot = kernel_dir = image_output_dir = "" > > if options.image_name: > (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot) = \ > @@ -140,34 +149,65 @@ def wic_create_subcommand(args, usage_str): > wks_file = args[0] > > if not wks_file.endswith(".wks"): > + # Return full path of the .wks file > wks_file = find_canned_image(scripts_path, wks_file) > if not wks_file: > - print "No image named %s found, exiting. (Use 'wic list images' to list available images, or specify a fully-qualified OE kickstart (.wks) filename)\n" % wks_file > + print "No image named %s found, exiting.\n" % wks_file > + print "(Use 'wic list images' to list available images, or specify a fully-qualified OE kickstart (.wks) filename)\n" > sys.exit(1) > > - image_output_dir = "" > if options.outdir: > image_output_dir = options.outdir > > - if not options.image_name: > - rootfs_dir = '' > - if 'ROOTFS_DIR' in options.rootfs_dir: > - rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > - bootimg_dir = options.bootimg_dir > - kernel_dir = options.kernel_dir > + if options.native_sysroot: > native_sysroot = options.native_sysroot > - if rootfs_dir and not os.path.isdir(rootfs_dir): > - print "--roofs-dir (-r) not found, exiting\n" > - sys.exit(1) > - if not os.path.isdir(bootimg_dir): > - print "--bootimg-dir (-b) not found, exiting\n" > - sys.exit(1) > - if not os.path.isdir(kernel_dir): > - print "--kernel-dir (-k) not found, exiting\n" > - sys.exit(1) > + print "Using native_sysroot from user command: %s" % native_sysroot > + > if not os.path.isdir(native_sysroot): > - print "--native-sysroot (-n) not found, exiting\n" > + print "--native-sysroot (-n) not found, exiting" > sys.exit(1) > + > + native_sysroot = os.path.abspath(native_sysroot) > + > + if not options.image_name: > + if (options.bootimg_dir and options.kernel_dir and > + options.rootfs_dir and options.native_sysroot): > + rootfs_dir = '' > + if 'ROOTFS_DIR' in options.rootfs_dir: > + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > + bootimg_dir = options.bootimg_dir > + kernel_dir = options.kernel_dir > + native_sysroot = options.native_sysroot > + > + if rootfs_dir and not os.path.isdir(rootfs_dir): > + print "--roofs-dir (-r) not found, exiting\n" > + sys.exit(1) > + if not os.path.isdir(bootimg_dir): > + print "--bootimg-dir (-b) not found, exiting\n" > + sys.exit(1) > + if not os.path.isdir(kernel_dir): > + print "--kernel-dir (-k) not found, exiting\n" > + sys.exit(1) > + if not os.path.isdir(native_sysroot): > + print "--native-sysroot (-n) not found, exiting\n" > + sys.exit(1) > + else: > + print 'Build image from rootfs and a package list using native rootfs\n' > + if options.rootfs_dir and 'ROOTFS_DIR' in options.rootfs_dir: > + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > + elif options.rootfs_dir: > + rootfs_dir = options.rootfs_dir > + else: > + rootfs_dir = "" > + > + native_sysroot = options.native_sysroot > + > + if rootfs_dir and not os.path.isdir(rootfs_dir): > + print "--roofs-dir (-r) not found, exiting\n" > + sys.exit(1) > + if not os.path.isdir(native_sysroot): > + print "--native-sysroot (-n) not found, exiting\n" > + sys.exit(1) > else: > not_found = not_found_dir = "" > if not os.path.isdir(rootfs_dir): > ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCHv2 0/2] wic: Add wic support to generate rootfs image for uboot 2014-06-12 2:49 ` Saul Wold @ 2014-06-12 8:43 ` y 2014-06-12 8:43 ` [PATCHv2 1/2] " y [not found] ` <53996893.88fd440a.0627.25d6SMTPIN_ADDED_BROKEN@mx.google.com> 1 sibling, 1 reply; 11+ messages in thread From: y @ 2014-06-12 8:43 UTC (permalink / raw) To: sgw, openembedded-core From: Adrian Calianu <adrian.calianu@enea.com> v2 Fix upstream compatibility with latest on master. v1 Hi, We tried to extend existing wic tool to generates rootfs images for U-Boot. This will be possible from cooked mode(existing build folder) and from raw mode where you have to provide a list of packages to be installed and a native rootfs with necessary tools. Some prerequisites are required for this new build setup in wic.conf file. As package manager only opkg and only web paths(repourl) are supported by now. ../poky/scripts/lib/image/config/wic.conf [create] arch=target_arch (example: powerpc, arm, x86) pkgmgr=opkg repourl=http://example.distro/p2020rdb/ipk/all http://example.distro/p2020rdb/ipk/p2020rdb http://example.distro/p2020rdb/ipk/ppce500v2 1) Build rootfs image from an existing bitbake build: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core-image-minimal -o .../output 2) Build rootfs image from an existing rootfs and native_sysroot: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r .../path_to_rootfs -n .../path_to_native_sysroot -o .../output 3) Build rootfs image only from a package list (on wks file) similar with IMAGE_INSTALL from bitbake. Package dependencies will be installed by default by the package manager. Add the list of packages to be installed as rootfs on ../poky/scripts/lib/image/canned-wks/uboot.wks: %packages packagegroup-core-boot pramfs-init run-postinsts packagegroup-core-ssh-dropbear %end Generate rootfs image: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n .../path_to_native_sysroot -o .../output Adrian Calianu (2): Add wic support to generate rootfs image for uboot Fix compatibility with upstream. scripts/lib/image/canned-wks/uboot.wks | 17 + scripts/lib/image/config/wic.conf | 4 + scripts/lib/mic/imager/direct.py | 4 + .../lib/mic/kickstart/custom_commands/partition.py | 128 ++++ scripts/lib/mic/pluginbase.py | 9 + scripts/lib/mic/plugins/source/uboot.py | 173 +++++ scripts/lib/mic/utils/oe/package_manager.py | 810 ++++++++++++++++++++ scripts/wic | 88 ++- 8 files changed, 1209 insertions(+), 24 deletions(-) create mode 100644 scripts/lib/image/canned-wks/uboot.wks create mode 100644 scripts/lib/mic/plugins/source/uboot.py create mode 100644 scripts/lib/mic/utils/oe/package_manager.py -- 1.7.10.4 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCHv2 1/2] Add wic support to generate rootfs image for uboot 2014-06-12 8:43 ` [PATCHv2 0/2] wic: " y @ 2014-06-12 8:43 ` y 2014-06-13 20:18 ` Tom Zanussi 0 siblings, 1 reply; 11+ messages in thread From: y @ 2014-06-12 8:43 UTC (permalink / raw) To: sgw, openembedded-core From: Adrian Calianu <adrian.calianu@enea.com> Add support in wic to build ramdisk uboot images from list of packages or from an existing rootfs folder. Providing the list of packages starting with XXX-core-boot package(into wks file) will generate an rootfs image for uboot out of bitbake. There is also possible to combine an existing/application rootfs with a list of packages that can be installed but the user must take care to provide a basic core boot rootfs or XXX-core-boot package and reporul(wic.conf). Some prerequisites are required for this new build setup: ../poky/scripts/lib/image/config/wic.conf [create] arch=target_arch (example: powerpc, arm, x86) pkgmgr=opkg repourl=http://example.distro/p2020rdb/ipk/all http://example.distro/p2020rdb/ipk/p2020rdb http://example.distro/p2020rdb/ipk/ppce500v2 1) Build an rootfs image from an existing bitbake build: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core-image-minimal -o .../output 2) Build an rootfs image from an existing rootfs and native_sysroot: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r .../path_to_rootfs -n .../path_to_native_sysroot -o .../output 3) Build an rootfs image only from a package list (on wks file): Add the package list to be installed as rootfs on ../poky/scripts/lib/image/canned-wks/uboot.wks: %packages packagegroup-core-boot pramfs-init run-postinsts packagegroup-core-ssh-dropbear %end Generate rootfs image: wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n .../path_to_native_sysroot -o .../output Signed-off-by: Adrian Calianu <adrian.calianu@enea.com> --- scripts/lib/image/canned-wks/uboot.wks | 17 + scripts/lib/image/config/wic.conf | 4 + scripts/lib/mic/imager/direct.py | 4 + .../lib/mic/kickstart/custom_commands/partition.py | 128 ++++ scripts/lib/mic/pluginbase.py | 9 + scripts/lib/mic/plugins/source/uboot.py | 173 +++++ scripts/lib/mic/utils/oe/package_manager.py | 810 ++++++++++++++++++++ scripts/wic | 88 ++- 10 files changed, 1219 insertions(+), 32 deletions(-) create mode 100644 scripts/lib/image/canned-wks/uboot.wks create mode 100644 scripts/lib/mic/plugins/source/uboot.py create mode 100644 scripts/lib/mic/utils/oe/package_manager.py diff --git a/scripts/lib/image/canned-wks/uboot.wks b/scripts/lib/image/canned-wks/uboot.wks new file mode 100644 index 0000000..7de0572 --- /dev/null +++ b/scripts/lib/image/canned-wks/uboot.wks @@ -0,0 +1,17 @@ +# short-description: . Create a ramdisk image for U-Boot +# long-description: Creates a ramdisk image for U-Boot that user +# can directly load it into ram through tftp +# +# part - is a wic command that drive the process of generating a valid file system +# - --source=uboot : wic plugin that generates a ramdisk image for U-Boot +# - --fstype=ext2 : file system type( ext2 / ext3 / ext 4) +# +# %packages %end - option to provide a list of packages that will be installed +# into rootfs. All packages dependencies will be installed by +# package manager(default opkg). + + +part / --source=uboot --fstype=ext2 --label imageName --align 1024 + +%packages +%end diff --git a/scripts/lib/image/config/wic.conf b/scripts/lib/image/config/wic.conf index e96d6ae..2a2750b 100644 --- a/scripts/lib/image/config/wic.conf +++ b/scripts/lib/image/config/wic.conf @@ -4,4 +4,8 @@ distro_name = OpenEmbedded [create] ; settings for create subcommand +; repourl=http://linux.com/ipk/all http://linux.com/ipk/target http://linux.com/ipk/arch +arch=powerpc +pkgmgr=opkg runtime=native +install_pkgs=source diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py index 2cf4c8d..fef9d0e 100644 --- a/scripts/lib/mic/imager/direct.py +++ b/scripts/lib/mic/imager/direct.py @@ -262,6 +262,10 @@ class DirectImageCreator(BaseImageCreator): # when/if we need to actually do package selection we # should modify things to use those objects, but for now # we can avoid that. + + p.install_pkgs(self, self.workdir, self.oe_builddir, self.rootfs_dir, + self.bootimg_dir, self.kernel_dir, self.native_sysroot) + p.prepare(self, self.workdir, self.oe_builddir, self.rootfs_dir, self.bootimg_dir, self.kernel_dir, self.native_sysroot) diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py index 6b575c0..450d2d4 100644 --- a/scripts/lib/mic/kickstart/custom_commands/partition.py +++ b/scripts/lib/mic/kickstart/custom_commands/partition.py @@ -31,7 +31,11 @@ from mic.utils.oe.misc import * from mic.kickstart.custom_commands import * from mic.plugin import pluginmgr +import os +from mic.utils.oe.package_manager import * + partition_methods = { + "do_install_pkgs":None, "do_stage_partition":None, "do_prepare_partition":None, "do_configure_partition":None, @@ -115,6 +119,102 @@ class Wic_PartData(Mic_PartData): else: return 0 + def install_pkgs(self, creator, cr_workdir, oe_builddir, rootfs_dir, + bootimg_dir, kernel_dir, native_sysroot): + """ + Prepare content for individual partitions, installing packages. + """ + + if not self.source: + return + + self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods) + self._source_methods["do_install_pkgs"](self, creator, + cr_workdir, + oe_builddir, + rootfs_dir, + bootimg_dir, + kernel_dir, + native_sysroot) + + def install_pkgs_ipk(self, cr_workdir, oe_builddir, rootfs_dir, + native_sysroot, packages, repourl): + """ + Install packages specified into wks file using opkg package manager. + This method is dependend on bb module. + """ + + gVar = {} + gVar["DEPLOY_DIR_IPK"] = os.path.join(oe_builddir, "tmp/deploy/ipk") + + # Run postinstall scripts even in offline mode + # Use the arch priority package rather than higher version one if more than one candidate is found. + #d.setVar("OPKG_ARGS", "--force_postinstall --prefer-arch-to-version") + gVar["OPKG_ARGS"] = "--force_postinstall" + + # OPKG path relative to /output_path + gVar["OPKGLIBDIR"] = "var/lib" + + source_url = repourl.split() + + # Generate feed uri's names, it doesn't seem to matter what name they have + feed_uris = "" + cnt = 0 + archs = "" + for url in source_url: + feed_uris += "cl_def_feed%d##%s\n" % (cnt, url) + cnt += 1 + head, tail = os.path.split(url) + archs += " " + tail + + # IPK_FEED_URIS with special formating defines the URI's used as source for packages + gVar['IPK_FEED_URIS'] = feed_uris + + gVar['BUILD_IMAGES_FROM_FEEDS'] = "1" + + # We need to provide sysroot for utilities + gVar['STAGING_DIR_NATIVE'] = native_sysroot + + # Set WORKDIR for output + gVar['WORKDIR'] = cr_workdir + + # Set TMPDIR for output + gVar['TMPDIR'] = os.path.join(cr_workdir, "tmp") + + if 'ROOTFS_DIR' in rootfs_dir: + target_dir = rootfs_dir['ROOTFS_DIR'] + elif os.path.isdir(rootfs_dir): + target_dir = rootfs_dir + else: + msg = "Couldn't find --rootfs-dir=%s connection" + msg += " or it is not a valid path, exiting" + msger.error(msg % rootfs_dir) + + # Need native sysroot /usr/bin/ for opkg-cl + # chnage PATH var to avoid issues with host tools + defpath = os.environ['PATH'] + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" + + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % target_dir + pseudo += "export PSEUDO_PASSWD=%s;" % target_dir + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "%s/usr/bin/pseudo " % native_sysroot + + pm = WicOpkgPM(gVar, + target_dir, + 'opkg.conf', + archs, + pseudo, + native_sysroot) + + pm.update() + + pm.install(packages) + + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" + + def prepare(self, cr, cr_workdir, oe_builddir, rootfs_dir, bootimg_dir, kernel_dir, native_sysroot): """ @@ -225,6 +325,34 @@ class Wic_PartData(Mic_PartData): return 0 + def prepare_for_uboot(self, arch, cr_workdir, oe_builddir, rootfs_dir, + native_sysroot): + """ + Generates u-boot image from source_file( ext2/3/4 ) + + """ + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" + pseudo += "%s/usr/bin/pseudo " % native_sysroot + + # 1) compress image + rootfs = self.source_file + rootfs_gzip = "%s.gz" % rootfs + gzip_cmd = "gzip -f -9 -c %s > %s" % (rootfs, rootfs_gzip) + rc, out = exec_native_cmd(pseudo + gzip_cmd, native_sysroot) + + # 2) image for U-Boot + rootfs_uboot = "%s.u-boot" % rootfs_gzip + mkimage_cmd = "mkimage -A %s -O linux -T ramdisk -C gzip -n %s -d %s %s" % \ + (arch, self.label, rootfs_gzip, rootfs_uboot) + rc, out = exec_native_cmd(pseudo + mkimage_cmd, native_sysroot) + + msger.info("\n\n\tThe new U-Boot ramdisk image can be found here:\n\t\t%s\n\n" % rootfs_uboot) + + return 0 + def prepare_rootfs_btrfs(self, cr_workdir, oe_builddir, rootfs_dir, native_sysroot, pseudo): """ diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py index 9cf4c62..881d996 100644 --- a/scripts/lib/mic/pluginbase.py +++ b/scripts/lib/mic/pluginbase.py @@ -89,6 +89,15 @@ class SourcePlugin(_Plugin): """ @classmethod + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, rootfs_dir, + bootimg_dir, kernel_dir, native_sysroot): + """ + Called before partitions have been prepared and assembled into a + disk image. Install packages into rootfs + """ + msger.debug("SourcePlugin: do_install_pkgs: part %s" % part) + + @classmethod def do_install_disk(self, disk, disk_name, cr, workdir, oe_builddir, bootimg_dir, kernel_dir, native_sysroot): """ diff --git a/scripts/lib/mic/plugins/source/uboot.py b/scripts/lib/mic/plugins/source/uboot.py new file mode 100644 index 0000000..57cb3cf --- /dev/null +++ b/scripts/lib/mic/plugins/source/uboot.py @@ -0,0 +1,173 @@ +# ex:ts=4:sw=4:sts=4:et +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- +# +# Copyright (c) 2014, Enea AB. +# All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# DESCRIPTION +# This implements the 'uboot' source plugin class for 'wic' +# +# AUTHORS +# Adrian Calianu <adrian.calianu (at] enea.com> +# + +import os +import shutil +import re +import tempfile + +from mic import kickstart, chroot, msger +from mic.utils import misc, fs_related, errors, runner, cmdln +from mic.conf import configmgr +from mic.plugin import pluginmgr +from mic.utils.partitionedfs import PartitionedMount +import mic.imager.direct as direct +from mic.pluginbase import SourcePlugin +from mic.utils.oe.misc import * +from mic.imager.direct import DirectImageCreator + +def create_local_rootfs(part, creator, cr_workdir, krootfs_dir, native_sysroot): + # In order to have a full control over rootfs we will make a local copy under workdir + # and change rootfs_dir to new location. + # In this way we can install more than one ROOTFS_DIRs and/or use + # an empty rootfs to install packages, so a rootfs could be generated only from pkgs + # TBD: create workdir/rootfs ; copy rootfs-> workdir/rootfs; set rootfs=workdir/rootfs + + cr_workdir = os.path.abspath(cr_workdir) + new_rootfs_dir = "%s/rootfs_%s" % (cr_workdir, creator.name) + + rootfs_exists = 1 + if part.rootfs is None: + if not 'ROOTFS_DIR' in krootfs_dir: + msg = "Couldn't find --rootfs-dir, exiting, " + msger.info(msg) + rootfs_exists = 0 + rootfs_dir = krootfs_dir['ROOTFS_DIR'] + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir + else: + if part.rootfs in krootfs_dir: + rootfs_dir = krootfs_dir[part.rootfs] + creator.rootfs_dir[part.rootfs] = new_rootfs_dir + elif os.path.isdir(part.rootfs): + rootfs_dir = part.rootfs + part.rootfs = new_rootfs_dir + else: + msg = "Couldn't find --rootfs-dir=%s connection" + msg += " or it is not a valid path, exiting" + msger.info(msg % part.rootfs) + rootfs_exists = 0 + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir + + pseudox = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudox += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % new_rootfs_dir + pseudox += "export PSEUDO_PASSWD=%s;" % new_rootfs_dir + pseudox += "export PSEUDO_NOSYMLINKEXP=1;" + pseudox += "%s/usr/bin/pseudo " % native_sysroot + + mkdir_cmd = "mkdir %s" % (new_rootfs_dir) + # rc, out = exec_native_cmd(pseudox + mkdir_cmd, native_sysroot) + rc, out = exec_cmd(mkdir_cmd, True) + + if rootfs_exists == 1 and os.path.isdir(rootfs_dir): + defpath = os.environ['PATH'] + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" + + rootfs_dir = os.path.abspath(rootfs_dir) + + pseudoc = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot + pseudoc += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir + pseudoc += "export PSEUDO_PASSWD=%s;" % rootfs_dir + pseudoc += "export PSEUDO_NOSYMLINKEXP=1;" + pseudoc += "%s/usr/bin/pseudo " % native_sysroot + + tarc_cmd = "tar cvpf %s/rootfs.tar -C %s ." % (cr_workdir, rootfs_dir) + rc, out = exec_native_cmd(pseudoc + tarc_cmd, native_sysroot) + + tarx_cmd = "tar xpvf %s/rootfs.tar -C %s" % (cr_workdir, new_rootfs_dir) + rc, out = exec_native_cmd(pseudox + tarx_cmd, native_sysroot) + + rm_cmd = "rm %s/rootfs.tar" % cr_workdir + rc, out = exec_cmd(rm_cmd, True) + + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" + + return new_rootfs_dir + +class UBootPlugin(SourcePlugin): + name = 'uboot' + + @classmethod + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, krootfs_dir, + bootimg_dir, kernel_dir, native_sysroot): + """ + Called before all partitions have been prepared and assembled into a + disk image. Intall packages based on wic configuration. + """ + + # set new rootfs_dir + rootfs_dir = create_local_rootfs(part, creator, cr_workdir, krootfs_dir, native_sysroot) + + # wks file parsing + packages = kickstart.get_packages(creator.ks) + + # wic.conf file parsing = found under 'creator' + local_pkgs_path = creator._local_pkgs_path + repourl = creator.repourl + pkgmgr = creator.pkgmgr_name + + # install packages + if packages and pkgmgr in ["opkg"]: + if len(repourl) > 0 : + part.install_pkgs_ipk(cr_workdir, oe_builddir, rootfs_dir, native_sysroot, + packages, repourl) + else: + msger.error("No packages repository provided in wic.conf") + + @classmethod + def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir, + kernel_dir, krootfs_dir, native_sysroot): + """ + Called to do the actual content population for a partition i.e. it + 'prepares' the partition to be incorporated into the image. + In this case, prepare content for legacy bios boot partition. + """ + if part.rootfs is None: + if not 'ROOTFS_DIR' in krootfs_dir: + msg = "Couldn't find --rootfs-dir, exiting" + msger.error(msg) + rootfs_dir = krootfs_dir['ROOTFS_DIR'] + else: + if part.rootfs in krootfs_dir: + rootfs_dir = krootfs_dir[part.rootfs] + elif os.path.isdir(part.rootfs): + rootfs_dir = part.rootfs + else: + msg = "Couldn't find --rootfs-dir=%s connection" + msg += " or it is not a valid path, exiting" + msger.error(msg % part.rootfs) + + part.set_rootfs(rootfs_dir) + + # change partition label wich will reflect into the final rootfs image name + part.label = "%s_%s" % (part.label, cr.name) + + defpath = os.environ['PATH'] + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" + + part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot) + part.prepare_for_uboot(cr.target_arch,cr_workdir, oe_builddir, rootfs_dir, native_sysroot) + + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" diff --git a/scripts/lib/mic/utils/oe/package_manager.py b/scripts/lib/mic/utils/oe/package_manager.py new file mode 100644 index 0000000..92ce98e --- /dev/null +++ b/scripts/lib/mic/utils/oe/package_manager.py @@ -0,0 +1,810 @@ +# ex:ts=4:sw=4:sts=4:et +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- +# +# Copyright (c) 2014, Enea AB. +# All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program; if not, write to the Free Software Foundation, Inc., +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# DESCRIPTION +# This implements the opkg package manager wrapper as a combination of +# meta/lib/oe/package_manager.py and bitbake/lib/bb/utils.py files and +# adaptation of those files to 'wic'. +# +# AUTHORS +# Adrian Calianu <adrian.calianu (at] enea.com> +# +# This file incorporates work covered by the following copyright and +# permission notice: +# +# meta/COPYING.GPLv2 (GPLv2) +# meta/COPYING.MIT (MIT) +# +# Copyright (C) 2004 Michael Lauer +# +# Permission to use, copy, modify, and/or distribute this software +# for any purpose with or without fee is hereby granted, provided +# that the above copyright notice and this permission notice appear +# in all copies. +# +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL +# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED +# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE +# AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR +# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS +# OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, +# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN +# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + +from abc import ABCMeta, abstractmethod +import os +import glob +import subprocess +import shutil +import multiprocessing +import re +import errno +import fcntl + +from mic.utils.oe.misc import * +from mic import msger + +def mkdirhier(directory): + """Create a directory like 'mkdir -p', but does not complain if + directory already exists like os.makedirs + """ + + try: + os.makedirs(directory) + except OSError as e: + if e.errno != errno.EEXIST: + raise e + +def remove(path, recurse=False): + """Equivalent to rm -f or rm -rf""" + if not path: + return + if recurse: + # shutil.rmtree(name) would be ideal but its too slow + subprocess.call(['rm', '-rf'] + glob.glob(path)) + return + for name in glob.glob(path): + try: + os.unlink(name) + except OSError as exc: + if exc.errno != errno.ENOENT: + raise + +def lockfile(name, shared=False, retry=True): + """ + Use the file fn as a lock file, return when the lock has been acquired. + Returns a variable to pass to unlockfile(). + """ + dirname = os.path.dirname(name) + mkdirhier(dirname) + + if not os.access(dirname, os.W_OK): + logger.error("Unable to acquire lock '%s', directory is not writable", + name) + sys.exit(1) + + op = fcntl.LOCK_EX + if shared: + op = fcntl.LOCK_SH + if not retry: + op = op | fcntl.LOCK_NB + + while True: + # If we leave the lockfiles lying around there is no problem + # but we should clean up after ourselves. This gives potential + # for races though. To work around this, when we acquire the lock + # we check the file we locked was still the lock file on disk. + # by comparing inode numbers. If they don't match or the lockfile + # no longer exists, we start again. + + # This implementation is unfair since the last person to request the + # lock is the most likely to win it. + + try: + lf = open(name, 'a+') + fileno = lf.fileno() + fcntl.flock(fileno, op) + statinfo = os.fstat(fileno) + if os.path.exists(lf.name): + statinfo2 = os.stat(lf.name) + if statinfo.st_ino == statinfo2.st_ino: + return lf + lf.close() + except Exception: + try: + lf.close() + except Exception: + pass + pass + if not retry: + return None + +def unlockfile(lf): + """ + Unlock a file locked using lockfile() + """ + try: + # If we had a shared lock, we need to promote to exclusive before + # removing the lockfile. Attempt this, ignore failures. + fcntl.flock(lf.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB) + os.unlink(lf.name) + except (IOError, OSError): + pass + fcntl.flock(lf.fileno(), fcntl.LOCK_UN) + lf.close() + +def which(path, item, direction = 0, history = False): + """ + Locate a file in a PATH + """ + + hist = [] + paths = (path or "").split(':') + if direction != 0: + paths.reverse() + + for p in paths: + next = os.path.join(p, item) + hist.append(next) + if os.path.exists(next): + if not os.path.isabs(next): + next = os.path.abspath(next) + if history: + return next, hist + return next + + if history: + return "", hist + return "" + + + +# this can be used by all PM backends to create the index files in parallel +def wic_create_index(arg): + index_cmd = arg + + try: + msger.info("Executing '%s' ..." % index_cmd) + subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True) + except subprocess.CalledProcessError as e: + return("Index creation command '%s' failed with return code %d:\n%s" % + (e.cmd, e.returncode, e.output)) + + return None + + +class WicIndexer(object): + __metaclass__ = ABCMeta + + def __init__(self, d, deploy_dir): + self.d = d + self.deploy_dir = deploy_dir + + @abstractmethod + def write_index(self): + pass + +class WicOpkgIndexer(WicIndexer): + def write_index(self): + arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS", + "SDK_PACKAGE_ARCHS", + "MULTILIB_ARCHS"] + + opkg_index_cmd = which(os.getenv('PATH'), "opkg-make-index") + + if not os.path.exists(os.path.join(self.deploy_dir, "Packages")): + open(os.path.join(self.deploy_dir, "Packages"), "w").close() + + index_cmds = [] + for arch_var in arch_vars: + if self.d.has_key(arch_var): + archs = self.d[arch_var] + else: + archs = None + + if archs is None: + continue + + for arch in archs.split(): + pkgs_dir = os.path.join(self.deploy_dir, arch) + pkgs_file = os.path.join(pkgs_dir, "Packages") + + if not os.path.isdir(pkgs_dir): + continue + + if not os.path.exists(pkgs_file): + open(pkgs_file, "w").close() + + index_cmds.append('%s -r %s -p %s -m %s' % + (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir)) + + if len(index_cmds) == 0: + msger.info("There are no packages in %s!" % self.deploy_dir) + return + + nproc = multiprocessing.cpu_count() + pool = multiprocessing.Pool(nproc) + results = list(pool.imap(wic_create_index, index_cmds)) + pool.close() + pool.join() + + for result in results: + if result is not None: + return(result) + +class WicPkgsList(object): + __metaclass__ = ABCMeta + + def __init__(self, d, rootfs_dir): + self.d = d + self.rootfs_dir = rootfs_dir + + @abstractmethod + def list(self, format=None): + pass + + +class WicOpkgPkgsList(WicPkgsList): + def __init__(self, d, rootfs_dir, config_file): + super(WicOpkgPkgsList, self).__init__(d, rootfs_dir) + + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") + self.opkg_args = "-f %s -o %s " % (config_file, rootfs_dir) + if self.d.has_key("OPKG_ARGS"): + self.opkg_args += self.d["OPKG_ARGS"] + + def list(self, format=None): + opkg_query_cmd = which(os.getenv('PATH'), "opkg-query-helper.py") + + if format == "arch": + cmd = "%s %s status | %s -a" % \ + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) + elif format == "file": + cmd = "%s %s status | %s -f" % \ + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) + elif format == "ver": + cmd = "%s %s status | %s -v" % \ + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) + elif format == "deps": + cmd = "%s %s status | %s" % \ + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) + else: + cmd = "%s %s list_installed | cut -d' ' -f1" % \ + (self.opkg_cmd, self.opkg_args) + + try: + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip() + except subprocess.CalledProcessError as e: + msger.error("Cannot get the installed packages list. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + if output and format == "file": + tmp_output = "" + for line in output.split('\n'): + pkg, pkg_file, pkg_arch = line.split() + full_path = os.path.join(self.rootfs_dir, pkg_arch, pkg_file) + if os.path.exists(full_path): + tmp_output += "%s %s %s\n" % (pkg, full_path, pkg_arch) + else: + tmp_output += "%s %s %s\n" % (pkg, pkg_file, pkg_arch) + + output = tmp_output + + return output + + +class WicPackageManager(object): + """ + This is an abstract class. Do not instantiate this directly. + """ + __metaclass__ = ABCMeta + + def __init__(self, d, pseudo, native_sysroot): + self.d = d + self.deploy_dir = None + self.deploy_lock = None + if self.d.has_key('PACKAGE_FEED_URIS'): + self.feed_uris = self.d['PACKAGE_FEED_URIS'] + else: + self.feed_uris = "" + self.pseudo = pseudo + self.native_sysroot = native_sysroot + + """ + Update the package manager package database. + """ + @abstractmethod + def update(self): + pass + + """ + Install a list of packages. 'pkgs' is a list object. If 'attempt_only' is + True, installation failures are ignored. + """ + @abstractmethod + def install(self, pkgs, attempt_only=False): + pass + + """ + Remove a list of packages. 'pkgs' is a list object. If 'with_dependencies' + is False, the any dependencies are left in place. + """ + @abstractmethod + def remove(self, pkgs, with_dependencies=True): + pass + + """ + This function creates the index files + """ + @abstractmethod + def write_index(self): + pass + + @abstractmethod + def remove_packaging_data(self): + pass + + @abstractmethod + def list_installed(self, format=None): + pass + + @abstractmethod + def insert_feeds_uris(self): + pass + + """ + Install complementary packages based upon the list of currently installed + packages e.g. locales, *-dev, *-dbg, etc. This will only attempt to install + these packages, if they don't exist then no error will occur. Note: every + backend needs to call this function explicitly after the normal package + installation + """ + def install_complementary(self, globs=None): + # we need to write the list of installed packages to a file because the + # oe-pkgdata-util reads it from a file + if self.d.has_key('WORKDIR'): + installed_pkgs_file = os.path.join(self.d['WORKDIR'], + "installed_pkgs.txt") + else: + msger.error("No WORKDIR provided!") + + with open(installed_pkgs_file, "w+") as installed_pkgs: + installed_pkgs.write(self.list_installed("arch")) + + if globs is None: + if self.d.has_key('IMAGE_INSTALL_COMPLEMENTARY'): + globs = self.d['IMAGE_INSTALL_COMPLEMENTARY'] + split_linguas = set() + + if self.d.has_key('IMAGE_LINGUAS'): + for translation in self.d['IMAGE_LINGUAS'].split(): + split_linguas.add(translation) + split_linguas.add(translation.split('-')[0]) + + split_linguas = sorted(split_linguas) + + for lang in split_linguas: + globs += " *-locale-%s" % lang + + if globs is None: + return + + if not self.d.has_key('PKGDATA_DIR'): + msger.error("No PKGDATA_DIR provided!") + + cmd = [which(os.getenv('PATH'), "oe-pkgdata-util"), + "glob", self.d['PKGDATA_DIR'], installed_pkgs_file, + globs] + + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) + if rc != 0: + msger.error("Could not compute complementary packages list. Command " + "'%s' returned %d" % + (' '.join(cmd), rc)) + + self.install(out.split(), attempt_only=True) + + + def deploy_dir_lock(self): + if self.deploy_dir is None: + raise RuntimeError("deploy_dir is not set!") + + lock_file_name = os.path.join(self.deploy_dir, "deploy.lock") + + self.deploy_lock = lockfile(lock_file_name) + + def deploy_dir_unlock(self): + if self.deploy_lock is None: + return + + unlockfile(self.deploy_lock) + + self.deploy_lock = None + + +class WicOpkgPM(WicPackageManager): + def __init__(self, d, target_rootfs, config_file, archs, pseudo, native_sysroot, task_name='target'): + super(WicOpkgPM, self).__init__(d, pseudo, native_sysroot) + + self.target_rootfs = target_rootfs + self.config_file = config_file + self.pkg_archs = archs + self.task_name = task_name + + if self.d.has_key("DEPLOY_DIR_IPK"): + self.deploy_dir = self.d["DEPLOY_DIR_IPK"] + + self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock") + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") + self.opkg_args = "-f %s -o %s " % (self.config_file, target_rootfs) + if self.d.has_key("OPKG_ARGS"): + self.opkg_args += self.d["OPKG_ARGS"] + + if self.d.has_key('OPKGLIBDIR'): + opkg_lib_dir = self.d['OPKGLIBDIR'] + else: + opkg_lib_dir = "" + + if opkg_lib_dir[0] == "/": + opkg_lib_dir = opkg_lib_dir[1:] + + self.opkg_dir = os.path.join(target_rootfs, opkg_lib_dir, "opkg") + + mkdirhier(self.opkg_dir) + + if self.d.has_key("TMPDIR"): + tmp_dir = self.d["TMPDIR"] + else: + tmp_dir = "" + + self.saved_opkg_dir = '%s/saved/%s' % (tmp_dir, self.task_name) + if not os.path.exists('%s/saved' % tmp_dir): + mkdirhier('%s/saved' % tmp_dir) + + if self.d.has_key('BUILD_IMAGES_FROM_FEEDS') and self.d['BUILD_IMAGES_FROM_FEEDS'] != "1": + self._create_config() + else: + self._create_custom_config() + + self.indexer = WicOpkgIndexer(self.d, self.deploy_dir) + + """ + This function will change a package's status in /var/lib/opkg/status file. + If 'packages' is None then the new_status will be applied to all + packages + """ + def mark_packages(self, status_tag, packages=None): + status_file = os.path.join(self.opkg_dir, "status") + + with open(status_file, "r") as sf: + with open(status_file + ".tmp", "w+") as tmp_sf: + if packages is None: + tmp_sf.write(re.sub(r"Package: (.*?)\n((?:[^\n]+\n)*?)Status: (.*)(?:unpacked|installed)", + r"Package: \1\n\2Status: \3%s" % status_tag, + sf.read())) + else: + if type(packages).__name__ != "list": + raise TypeError("'packages' should be a list object") + + status = sf.read() + for pkg in packages: + status = re.sub(r"Package: %s\n((?:[^\n]+\n)*?)Status: (.*)(?:unpacked|installed)" % pkg, + r"Package: %s\n\1Status: \2%s" % (pkg, status_tag), + status) + + tmp_sf.write(status) + + os.rename(status_file + ".tmp", status_file) + + def _create_custom_config(self): + msger.info("Building from feeds activated!") + + with open(self.config_file, "w+") as config_file: + priority = 1 + for arch in self.pkg_archs.split(): + config_file.write("arch %s %d\n" % (arch, priority)) + priority += 5 + + if self.d.has_key('IPK_FEED_URIS'): + ipk_feed_uris = self.d['IPK_FEED_URIS'] + else: + ipk_feed_uris = "" + + for line in ipk_feed_uris.split(): + feed_match = re.match("^[ \t]*(.*)##([^ \t]*)[ \t]*$", line) + + if feed_match is not None: + feed_name = feed_match.group(1) + feed_uri = feed_match.group(2) + + msger.info("Add %s feed with URL %s" % (feed_name, feed_uri)) + + config_file.write("src/gz %s %s\n" % (feed_name, feed_uri)) + + """ + Allow to use package deploy directory contents as quick devel-testing + feed. This creates individual feed configs for each arch subdir of those + specified as compatible for the current machine. + NOTE: Development-helper feature, NOT a full-fledged feed. + """ + if self.d.has_key('FEED_DEPLOYDIR_BASE_URI'): + feed_deploydir_base_dir = self.d['FEED_DEPLOYDIR_BASE_URI'] + else: + feed_deploydir_base_dir = "" + + if feed_deploydir_base_dir != "": + for arch in self.pkg_archs.split(): + if self.d.has_key("sysconfdir"): + sysconfdir = self.d["sysconfdir"] + else: + sysconfdir = None + + cfg_file_name = os.path.join(self.target_rootfs, + sysconfdir, + "opkg", + "local-%s-feed.conf" % arch) + + with open(cfg_file_name, "w+") as cfg_file: + cfg_file.write("src/gz local-%s %s/%s" % + arch, + feed_deploydir_base_dir, + arch) + + def _create_config(self): + with open(self.config_file, "w+") as config_file: + priority = 1 + for arch in self.pkg_archs.split(): + config_file.write("arch %s %d\n" % (arch, priority)) + priority += 5 + + config_file.write("src oe file:%s\n" % self.deploy_dir) + + for arch in self.pkg_archs.split(): + pkgs_dir = os.path.join(self.deploy_dir, arch) + if os.path.isdir(pkgs_dir): + config_file.write("src oe-%s file:%s\n" % + (arch, pkgs_dir)) + + def insert_feeds_uris(self): + if self.feed_uris == "": + return + + rootfs_config = os.path.join('%s/etc/opkg/base-feeds.conf' + % self.target_rootfs) + + with open(rootfs_config, "w+") as config_file: + uri_iterator = 0 + for uri in self.feed_uris.split(): + config_file.write("src/gz url-%d %s/ipk\n" % + (uri_iterator, uri)) + + for arch in self.pkg_archs.split(): + if not os.path.exists(os.path.join(self.deploy_dir, arch)): + continue + msger.info('Note: adding opkg channel url-%s-%d (%s)' % + (arch, uri_iterator, uri)) + + config_file.write("src/gz uri-%s-%d %s/ipk/%s\n" % + (arch, uri_iterator, uri, arch)) + uri_iterator += 1 + + def update(self): + self.deploy_dir_lock() + + cmd = "%s %s update" % (self.opkg_cmd, self.opkg_args) + + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) + if rc != 0: + self.deploy_dir_unlock() + msger.error("Unable to update the package index files. Command '%s' " + "returned %d" % (cmd, rc)) + + self.deploy_dir_unlock() + + def install(self, pkgs, attempt_only=False): + if attempt_only and len(pkgs) == 0: + return + + cmd = "%s %s install %s" % (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) + + os.environ['D'] = self.target_rootfs + os.environ['OFFLINE_ROOT'] = self.target_rootfs + os.environ['IPKG_OFFLINE_ROOT'] = self.target_rootfs + os.environ['OPKG_OFFLINE_ROOT'] = self.target_rootfs + if self.d.has_key('WORKDIR'): + os.environ['INTERCEPT_DIR'] = os.path.join(self.d['WORKDIR'], + "intercept_scripts") + else: + os.environ['INTERCEPT_DIR'] = "." + msger.warning("No WORKDIR provided!") + + if self.d.has_key('STAGING_DIR_NATIVE'): + os.environ['NATIVE_ROOT'] = self.d['STAGING_DIR_NATIVE'] + else: + msger.error("No STAGING_DIR_NATIVE provided!") + + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) + if rc != 0: + msger.error("Unable to install packages. " + "Command '%s' returned %d" % (cmd, rc)) + + + def remove(self, pkgs, with_dependencies=True): + if with_dependencies: + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \ + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) + else: + cmd = "%s %s --force-depends remove %s" % \ + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) + + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) + if rc != 0: + msger.error("Unable to remove packages. Command '%s' " + "returned %d" % (cmd, rc)) + + + def write_index(self): + self.deploy_dir_lock() + + result = self.indexer.write_index() + + self.deploy_dir_unlock() + + if result is not None: + msger.error(result) + + def remove_packaging_data(self): + remove(self.opkg_dir, True) + # create the directory back, it's needed by PM lock + mkdirhier(self.opkg_dir) + + def list_installed(self, format=None): + return WicOpkgPkgsList(self.d, self.target_rootfs, self.config_file).list(format) + + def handle_bad_recommendations(self): + if self.d.has_key("BAD_RECOMMENDATIONS"): + bad_recommendations = self.d["BAD_RECOMMENDATIONS"] + else: + bad_recommendations = "" + + if bad_recommendations.strip() == "": + return + + status_file = os.path.join(self.opkg_dir, "status") + + # If status file existed, it means the bad recommendations has already + # been handled + if os.path.exists(status_file): + return + + cmd = "%s %s info " % (self.opkg_cmd, self.opkg_args) + + with open(status_file, "w+") as status: + for pkg in bad_recommendations.split(): + pkg_info = cmd + pkg + + try: + output = subprocess.check_output(pkg_info.split(), stderr=subprocess.STDOUT).strip() + except subprocess.CalledProcessError as e: + msger.error("Cannot get package info. Command '%s' " + "returned %d:\n%s" % (pkg_info, e.returncode, e.output)) + + if output == "": + msger.info("Ignored bad recommendation: '%s' is " + "not a package" % pkg) + continue + + for line in output.split('\n'): + if line.startswith("Status:"): + status.write("Status: deinstall hold not-installed\n") + else: + status.write(line + "\n") + + ''' + The following function dummy installs pkgs and returns the log of output. + ''' + def dummy_install(self, pkgs): + if len(pkgs) == 0: + return + + # Create an temp dir as opkg root for dummy installation + if self.d.has_key("TMPDIR"): + tmp_dir = self.d["TMPDIR"] + else: + tmp_dir = "." + msger.warning("No TMPDIR provided!") + + temp_rootfs = '%s/opkg' % tmp_dir + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') + mkdirhier(temp_opkg_dir) + + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) + if self.d.has_key("OPKG_ARGS"): + opkg_args += self.d["OPKG_ARGS"] + + cmd = "%s %s update" % (self.opkg_cmd, opkg_args) + try: + subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) + except subprocess.CalledProcessError as e: + msger.error("Unable to update. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + # Dummy installation + cmd = "%s %s --noaction install %s " % (self.opkg_cmd, + opkg_args, + ' '.join(pkgs)) + try: + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) + except subprocess.CalledProcessError as e: + msger.error("Unable to dummy install packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + remove(temp_rootfs, True) + + return output + + def backup_packaging_data(self): + # Save the opkglib for increment ipk image generation + if os.path.exists(self.saved_opkg_dir): + remove(self.saved_opkg_dir, True) + shutil.copytree(self.opkg_dir, + self.saved_opkg_dir, + symlinks=True) + + def recover_packaging_data(self): + # Move the opkglib back + if os.path.exists(self.saved_opkg_dir): + if os.path.exists(self.opkg_dir): + remove(self.opkg_dir, True) + + msger.info('Recover packaging data') + shutil.copytree(self.saved_opkg_dir, + self.opkg_dir, + symlinks=True) + + +def wic_generate_index_files(d): + if d.has_key('PACKAGE_CLASSES'): + classes = d['PACKAGE_CLASSES'].replace("package_", "").split() + else: + classes = "" + msger.warning("No PACKAGE_CLASSES provided!") + + if d.has_key('DEPLOY_DIR_IPK'): + deploy_dir_ipk = d['DEPLOY_DIR_IPK'] + else: + deploy_dir_ipk = None + msger.warning("No DEPLOY_DIR_IPK provided!") + + indexer_map = { + "ipk": (WicOpkgIndexer, deploy_dir_ipk) + } + + result = None + + for pkg_class in classes: + if not pkg_class in indexer_map: + continue + + if os.path.exists(indexer_map[pkg_class][1]): + result = indexer_map[pkg_class][0](d, indexer_map[pkg_class][1]).write_index() + + if result is not None: + msger.error(result) diff --git a/scripts/wic b/scripts/wic index 2d3fd09..b9c8756 100755 --- a/scripts/wic +++ b/scripts/wic @@ -99,6 +99,10 @@ def wic_create_subcommand(args, usage_str): (options, args) = parser.parse_args(args) + if options.debug: + loglevel = logging.DEBUG + start_logging(loglevel) + if len(args) != 1: logging.error("Wrong number of arguments, exiting\n") parser.print_help() @@ -107,9 +111,11 @@ def wic_create_subcommand(args, usage_str): if not options.image_name and not (options.rootfs_dir and options.bootimg_dir and options.kernel_dir and + options.native_sysroot or options.native_sysroot): print "Build artifacts not completely specified, exiting." - print " (Use 'wic -e' or 'wic -r -b -k -n' to specify artifacts)" + print " (Use 'wic -e' or 'wic -r -b -k -n' or 'wic -r -n' to specify artifacts)" + print options sys.exit(1) if not options.image_name: @@ -125,13 +131,16 @@ def wic_create_subcommand(args, usage_str): print "Creating image(s)...\n" - bitbake_env_lines = find_bitbake_env_lines(options.image_name) - if not bitbake_env_lines: - print "Couldn't get bitbake environment, exiting." - sys.exit(1) - set_bitbake_env_lines(bitbake_env_lines) + # If '-e' option is used the values are extracted from bitbake env. + if options.image_name: + bitbake_env_lines = find_bitbake_env_lines(options.image_name) + if not bitbake_env_lines: + print "Couldn't get bitbake environment, exiting." + sys.exit(1) + set_bitbake_env_lines(bitbake_env_lines) bootimg_dir = staging_data_dir = hdddir = "" + rootfs_dir = native_sysroot = kernel_dir = image_output_dir = "" if options.image_name: (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot) = \ @@ -140,34 +149,65 @@ def wic_create_subcommand(args, usage_str): wks_file = args[0] if not wks_file.endswith(".wks"): + # Return full path of the .wks file wks_file = find_canned_image(scripts_path, wks_file) if not wks_file: - print "No image named %s found, exiting. (Use 'wic list images' to list available images, or specify a fully-qualified OE kickstart (.wks) filename)\n" % wks_file + print "No image named %s found, exiting.\n" % wks_file + print "(Use 'wic list images' to list available images, or specify a fully-qualified OE kickstart (.wks) filename)\n" sys.exit(1) - image_output_dir = "" if options.outdir: image_output_dir = options.outdir - if not options.image_name: - rootfs_dir = '' - if 'ROOTFS_DIR' in options.rootfs_dir: - rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] - bootimg_dir = options.bootimg_dir - kernel_dir = options.kernel_dir + if options.native_sysroot: native_sysroot = options.native_sysroot - if rootfs_dir and not os.path.isdir(rootfs_dir): - print "--roofs-dir (-r) not found, exiting\n" - sys.exit(1) - if not os.path.isdir(bootimg_dir): - print "--bootimg-dir (-b) not found, exiting\n" - sys.exit(1) - if not os.path.isdir(kernel_dir): - print "--kernel-dir (-k) not found, exiting\n" - sys.exit(1) + print "Using native_sysroot from user command: %s" % native_sysroot + if not os.path.isdir(native_sysroot): - print "--native-sysroot (-n) not found, exiting\n" + print "--native-sysroot (-n) not found, exiting" sys.exit(1) + + native_sysroot = os.path.abspath(native_sysroot) + + if not options.image_name: + if (options.bootimg_dir and options.kernel_dir and + options.rootfs_dir and options.native_sysroot): + rootfs_dir = '' + if 'ROOTFS_DIR' in options.rootfs_dir: + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] + bootimg_dir = options.bootimg_dir + kernel_dir = options.kernel_dir + native_sysroot = options.native_sysroot + + if rootfs_dir and not os.path.isdir(rootfs_dir): + print "--roofs-dir (-r) not found, exiting\n" + sys.exit(1) + if not os.path.isdir(bootimg_dir): + print "--bootimg-dir (-b) not found, exiting\n" + sys.exit(1) + if not os.path.isdir(kernel_dir): + print "--kernel-dir (-k) not found, exiting\n" + sys.exit(1) + if not os.path.isdir(native_sysroot): + print "--native-sysroot (-n) not found, exiting\n" + sys.exit(1) + else: + print 'Build image from rootfs and a package list using native rootfs\n' + if options.rootfs_dir and 'ROOTFS_DIR' in options.rootfs_dir: + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] + elif options.rootfs_dir: + rootfs_dir = options.rootfs_dir + else: + rootfs_dir = "" + + native_sysroot = options.native_sysroot + + if rootfs_dir and not os.path.isdir(rootfs_dir): + print "--roofs-dir (-r) not found, exiting\n" + sys.exit(1) + if not os.path.isdir(native_sysroot): + print "--native-sysroot (-n) not found, exiting\n" + sys.exit(1) else: not_found = not_found_dir = "" if not os.path.isdir(rootfs_dir): -- 1.7.10.4 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCHv2 1/2] Add wic support to generate rootfs image for uboot 2014-06-12 8:43 ` [PATCHv2 1/2] " y @ 2014-06-13 20:18 ` Tom Zanussi 2014-06-16 13:18 ` Adrian Calianu 0 siblings, 1 reply; 11+ messages in thread From: Tom Zanussi @ 2014-06-13 20:18 UTC (permalink / raw) To: y; +Cc: openembedded-core, Otavio Salvador Hi, This looks like some nice new functionality, but I have some suggestions regarding the implementation and the posting itself... First, this seems to have been sent by a 'y' at enea, but they're apparently written by adrian.calianu at enea - why can't adrian.calianu send them himself? (makes me wonder about the authorship, etc). Also, there seems to be just one big patch, but the subject says '[PATCHv2 1/2] Add wic support to generate rootfs image for uboot' - am I missing 2/2? As for the patch itself, it's huge and needs to be broken up into smaller pieces. In fact, it seems to be a combination of two completely separate things at least - a new mechanism for creating a rootfs from packages, and a new plugin for creating uboot partitions (which just happens to use the output from the new packaging step. So at minimum I think you need to break it up along the lines of that functionality, and it would be good to break it up even further e.g. 'Add a new do_pkg interface hook', 'Add an implementation of do_pkg for ipk', 'Add a new source plugin for uboot partitions', 'Add a new canned wks for a default uboot image', etc. The more you can break it up into logical chunks without going overboard, the easier it is to review and apply/revert. Finally, something like this really needs to be documented in the tool - please add the relevant interface documentation to the help system. For the 'packaging' part of this patch, the big problem I see is that the package_manager seems to be basically a copy of the oe package manager and more (which you mention in the comments) - I don't think it makes much sense to do that - wouldn't it be better to patch the oe package_manager code so that it could be reused by this tool? For the 'uboot' part, is the 'uboot' plugin implementing common functionality that all uboot images need, or is it for a specific use case (if so, it shouldn't be called 'uboot' but something more specific)? Otavio seemed to suggest that in general it would need to be modified for specific cases. I'm not sure about the best way to do that - we don't have many good examples yet. Can that sort of thing be parameterized/subclassed, or would it require a per-image plugin, which would be good to avoid. Not something you have to solve now, if the 'uboot' plugin is useful on its own and maybe subclassed/added onto by future plugins. It would be nice to have some specific and succinct examples to help think through what might be needed here. Some more specific comments in-line below... On Thu, 2014-06-12 at 10:43 +0200, y@enea.se wrote: > From: Adrian Calianu <adrian.calianu@enea.com> > > Add support in wic to build ramdisk uboot images from list of packages or > from an existing rootfs folder. > > Providing the list of packages starting with XXX-core-boot package(into wks file) > will generate an rootfs image for uboot out of bitbake. > There is also possible to combine an existing/application rootfs with a list > of packages that can be installed but the user must take care to provide > a basic core boot rootfs or XXX-core-boot package and reporul(wic.conf). > Is this implemented in the current patch? > Some prerequisites are required for this new build setup: > ../poky/scripts/lib/image/config/wic.conf > [create] > arch=target_arch (example: powerpc, arm, x86) > pkgmgr=opkg > repourl=http://example.distro/p2020rdb/ipk/all http://example.distro/p2020rdb/ipk/p2020rdb http://example.distro/p2020rdb/ipk/ppce500v2 > > 1) Build an rootfs image from an existing bitbake build: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core-image-minimal -o .../output > > 2) Build an rootfs image from an existing rootfs and native_sysroot: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r .../path_to_rootfs -n .../path_to_native_sysroot -o .../output > > 3) Build an rootfs image only from a package list (on wks file): > > Add the package list to be installed as rootfs on ../poky/scripts/lib/image/canned-wks/uboot.wks: > %packages > packagegroup-core-boot > pramfs-init > run-postinsts > packagegroup-core-ssh-dropbear > %end > > Generate rootfs image: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n .../path_to_native_sysroot -o .../output > > Signed-off-by: Adrian Calianu <adrian.calianu@enea.com> > --- > scripts/lib/image/canned-wks/uboot.wks | 17 + > scripts/lib/image/config/wic.conf | 4 + > scripts/lib/mic/imager/direct.py | 4 + > .../lib/mic/kickstart/custom_commands/partition.py | 128 ++++ > scripts/lib/mic/pluginbase.py | 9 + > scripts/lib/mic/plugins/source/uboot.py | 173 +++++ > scripts/lib/mic/utils/oe/package_manager.py | 810 ++++++++++++++++++++ > scripts/wic | 88 ++- > 10 files changed, 1219 insertions(+), 32 deletions(-) > create mode 100644 scripts/lib/image/canned-wks/uboot.wks > create mode 100644 scripts/lib/mic/plugins/source/uboot.py > create mode 100644 scripts/lib/mic/utils/oe/package_manager.py > > diff --git a/scripts/lib/image/canned-wks/uboot.wks b/scripts/lib/image/canned-wks/uboot.wks > new file mode 100644 > index 0000000..7de0572 > --- /dev/null > +++ b/scripts/lib/image/canned-wks/uboot.wks > @@ -0,0 +1,17 @@ > +# short-description: . Create a ramdisk image for U-Boot > +# long-description: Creates a ramdisk image for U-Boot that user > +# can directly load it into ram through tftp > +# > +# part - is a wic command that drive the process of generating a valid file system > +# - --source=uboot : wic plugin that generates a ramdisk image for U-Boot > +# - --fstype=ext2 : file system type( ext2 / ext3 / ext 4) > +# > +# %packages %end - option to provide a list of packages that will be installed > +# into rootfs. All packages dependencies will be installed by > +# package manager(default opkg). > + > + > +part / --source=uboot --fstype=ext2 --label imageName --align 1024 > + > +%packages > +%end > diff --git a/scripts/lib/image/config/wic.conf b/scripts/lib/image/config/wic.conf > index e96d6ae..2a2750b 100644 > --- a/scripts/lib/image/config/wic.conf > +++ b/scripts/lib/image/config/wic.conf > @@ -4,4 +4,8 @@ distro_name = OpenEmbedded > > [create] > ; settings for create subcommand > +; repourl=http://linux.com/ipk/all http://linux.com/ipk/target http://linux.com/ipk/arch > +arch=powerpc > +pkgmgr=opkg > runtime=native > +install_pkgs=source I don't think you want these settings as defaults in wic.conf. Maybe in the documentation? > diff --git a/scripts/lib/mic/imager/direct.py b/scripts/lib/mic/imager/direct.py > index 2cf4c8d..fef9d0e 100644 > --- a/scripts/lib/mic/imager/direct.py > +++ b/scripts/lib/mic/imager/direct.py > @@ -262,6 +262,10 @@ class DirectImageCreator(BaseImageCreator): > # when/if we need to actually do package selection we > # should modify things to use those objects, but for now > # we can avoid that. > + > + p.install_pkgs(self, self.workdir, self.oe_builddir, self.rootfs_dir, > + self.bootimg_dir, self.kernel_dir, self.native_sysroot) > + > p.prepare(self, self.workdir, self.oe_builddir, self.rootfs_dir, > self.bootimg_dir, self.kernel_dir, self.native_sysroot) > > diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py > index 6b575c0..450d2d4 100644 > --- a/scripts/lib/mic/kickstart/custom_commands/partition.py > +++ b/scripts/lib/mic/kickstart/custom_commands/partition.py > @@ -31,7 +31,11 @@ from mic.utils.oe.misc import * > from mic.kickstart.custom_commands import * > from mic.plugin import pluginmgr > > +import os > +from mic.utils.oe.package_manager import * > + > partition_methods = { > + "do_install_pkgs":None, > "do_stage_partition":None, > "do_prepare_partition":None, > "do_configure_partition":None, > @@ -115,6 +119,102 @@ class Wic_PartData(Mic_PartData): > else: > return 0 > > + def install_pkgs(self, creator, cr_workdir, oe_builddir, rootfs_dir, > + bootimg_dir, kernel_dir, native_sysroot): > + """ > + Prepare content for individual partitions, installing packages. > + """ > + > + if not self.source: > + return > + > + self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods) > + self._source_methods["do_install_pkgs"](self, creator, > + cr_workdir, > + oe_builddir, > + rootfs_dir, > + bootimg_dir, > + kernel_dir, > + native_sysroot) > + > + def install_pkgs_ipk(self, cr_workdir, oe_builddir, rootfs_dir, > + native_sysroot, packages, repourl): > + """ > + Install packages specified into wks file using opkg package manager. > + This method is dependend on bb module. > + """ > + > + gVar = {} > + gVar["DEPLOY_DIR_IPK"] = os.path.join(oe_builddir, "tmp/deploy/ipk") > + > + # Run postinstall scripts even in offline mode > + # Use the arch priority package rather than higher version one if more than one candidate is found. > + #d.setVar("OPKG_ARGS", "--force_postinstall --prefer-arch-to-version") > + gVar["OPKG_ARGS"] = "--force_postinstall" > + > + # OPKG path relative to /output_path > + gVar["OPKGLIBDIR"] = "var/lib" > + > + source_url = repourl.split() > + > + # Generate feed uri's names, it doesn't seem to matter what name they have > + feed_uris = "" > + cnt = 0 > + archs = "" > + for url in source_url: > + feed_uris += "cl_def_feed%d##%s\n" % (cnt, url) > + cnt += 1 > + head, tail = os.path.split(url) > + archs += " " + tail > + > + # IPK_FEED_URIS with special formating defines the URI's used as source for packages > + gVar['IPK_FEED_URIS'] = feed_uris > + > + gVar['BUILD_IMAGES_FROM_FEEDS'] = "1" > + > + # We need to provide sysroot for utilities > + gVar['STAGING_DIR_NATIVE'] = native_sysroot > + > + # Set WORKDIR for output > + gVar['WORKDIR'] = cr_workdir > + > + # Set TMPDIR for output > + gVar['TMPDIR'] = os.path.join(cr_workdir, "tmp") > + > + if 'ROOTFS_DIR' in rootfs_dir: > + target_dir = rootfs_dir['ROOTFS_DIR'] > + elif os.path.isdir(rootfs_dir): > + target_dir = rootfs_dir > + else: > + msg = "Couldn't find --rootfs-dir=%s connection" > + msg += " or it is not a valid path, exiting" > + msger.error(msg % rootfs_dir) > + > + # Need native sysroot /usr/bin/ for opkg-cl > + # chnage PATH var to avoid issues with host tools > + defpath = os.environ['PATH'] > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > + > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % target_dir > + pseudo += "export PSEUDO_PASSWD=%s;" % target_dir > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudo += "%s/usr/bin/pseudo " % native_sysroot > + > + pm = WicOpkgPM(gVar, > + target_dir, > + 'opkg.conf', > + archs, > + pseudo, > + native_sysroot) > + > + pm.update() > + > + pm.install(packages) > + > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > + > + > def prepare(self, cr, cr_workdir, oe_builddir, rootfs_dir, bootimg_dir, > kernel_dir, native_sysroot): > """ > @@ -225,6 +325,34 @@ class Wic_PartData(Mic_PartData): > > return 0 > > + def prepare_for_uboot(self, arch, cr_workdir, oe_builddir, rootfs_dir, > + native_sysroot): > + """ > + Generates u-boot image from source_file( ext2/3/4 ) > + > + """ > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir > + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudo += "%s/usr/bin/pseudo " % native_sysroot > + > + # 1) compress image > + rootfs = self.source_file > + rootfs_gzip = "%s.gz" % rootfs > + gzip_cmd = "gzip -f -9 -c %s > %s" % (rootfs, rootfs_gzip) > + rc, out = exec_native_cmd(pseudo + gzip_cmd, native_sysroot) > + > + # 2) image for U-Boot > + rootfs_uboot = "%s.u-boot" % rootfs_gzip > + mkimage_cmd = "mkimage -A %s -O linux -T ramdisk -C gzip -n %s -d %s %s" % \ > + (arch, self.label, rootfs_gzip, rootfs_uboot) > + rc, out = exec_native_cmd(pseudo + mkimage_cmd, native_sysroot) > + > + msger.info("\n\n\tThe new U-Boot ramdisk image can be found here:\n\t\t%s\n\n" % rootfs_uboot) > + > + return 0 > + > def prepare_rootfs_btrfs(self, cr_workdir, oe_builddir, rootfs_dir, > native_sysroot, pseudo): > """ > diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py > index 9cf4c62..881d996 100644 > --- a/scripts/lib/mic/pluginbase.py > +++ b/scripts/lib/mic/pluginbase.py > @@ -89,6 +89,15 @@ class SourcePlugin(_Plugin): > """ > > @classmethod > + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, rootfs_dir, > + bootimg_dir, kernel_dir, native_sysroot): > + """ > + Called before partitions have been prepared and assembled into a > + disk image. Install packages into rootfs > + """ > + msger.debug("SourcePlugin: do_install_pkgs: part %s" % part) > + > + @classmethod > def do_install_disk(self, disk, disk_name, cr, workdir, oe_builddir, > bootimg_dir, kernel_dir, native_sysroot): > """ > diff --git a/scripts/lib/mic/plugins/source/uboot.py b/scripts/lib/mic/plugins/source/uboot.py > new file mode 100644 > index 0000000..57cb3cf > --- /dev/null > +++ b/scripts/lib/mic/plugins/source/uboot.py > @@ -0,0 +1,173 @@ > +# ex:ts=4:sw=4:sts=4:et > +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > +# > +# Copyright (c) 2014, Enea AB. > +# All rights reserved. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License version 2 as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > +# > +# DESCRIPTION > +# This implements the 'uboot' source plugin class for 'wic' > +# > +# AUTHORS > +# Adrian Calianu <adrian.calianu (at] enea.com> > +# > + > +import os > +import shutil > +import re > +import tempfile > + > +from mic import kickstart, chroot, msger > +from mic.utils import misc, fs_related, errors, runner, cmdln > +from mic.conf import configmgr > +from mic.plugin import pluginmgr > +from mic.utils.partitionedfs import PartitionedMount > +import mic.imager.direct as direct > +from mic.pluginbase import SourcePlugin > +from mic.utils.oe.misc import * > +from mic.imager.direct import DirectImageCreator > + > +def create_local_rootfs(part, creator, cr_workdir, krootfs_dir, native_sysroot): > + # In order to have a full control over rootfs we will make a local copy under workdir > + # and change rootfs_dir to new location. > + # In this way we can install more than one ROOTFS_DIRs and/or use > + # an empty rootfs to install packages, so a rootfs could be generated only from pkgs > + # TBD: create workdir/rootfs ; copy rootfs-> workdir/rootfs; set rootfs=workdir/rootfs > + Is this something that would be needed by any image creating a rootfs from packages? If so, it should be part of the packaging functionality, not the uboot implementation specifically. > + cr_workdir = os.path.abspath(cr_workdir) > + new_rootfs_dir = "%s/rootfs_%s" % (cr_workdir, creator.name) > + > + rootfs_exists = 1 > + if part.rootfs is None: > + if not 'ROOTFS_DIR' in krootfs_dir: > + msg = "Couldn't find --rootfs-dir, exiting, " > + msger.info(msg) > + rootfs_exists = 0 > + rootfs_dir = krootfs_dir['ROOTFS_DIR'] > + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir > + else: > + if part.rootfs in krootfs_dir: > + rootfs_dir = krootfs_dir[part.rootfs] > + creator.rootfs_dir[part.rootfs] = new_rootfs_dir > + elif os.path.isdir(part.rootfs): > + rootfs_dir = part.rootfs > + part.rootfs = new_rootfs_dir > + else: > + msg = "Couldn't find --rootfs-dir=%s connection" > + msg += " or it is not a valid path, exiting" > + msger.info(msg % part.rootfs) > + rootfs_exists = 0 > + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir > + > + pseudox = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudox += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % new_rootfs_dir > + pseudox += "export PSEUDO_PASSWD=%s;" % new_rootfs_dir > + pseudox += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudox += "%s/usr/bin/pseudo " % native_sysroot > + > + mkdir_cmd = "mkdir %s" % (new_rootfs_dir) > + # rc, out = exec_native_cmd(pseudox + mkdir_cmd, native_sysroot) > + rc, out = exec_cmd(mkdir_cmd, True) > + > + if rootfs_exists == 1 and os.path.isdir(rootfs_dir): > + defpath = os.environ['PATH'] > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > + > + rootfs_dir = os.path.abspath(rootfs_dir) > + > + pseudoc = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > + pseudoc += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir > + pseudoc += "export PSEUDO_PASSWD=%s;" % rootfs_dir > + pseudoc += "export PSEUDO_NOSYMLINKEXP=1;" > + pseudoc += "%s/usr/bin/pseudo " % native_sysroot > + > + tarc_cmd = "tar cvpf %s/rootfs.tar -C %s ." % (cr_workdir, rootfs_dir) > + rc, out = exec_native_cmd(pseudoc + tarc_cmd, native_sysroot) > + > + tarx_cmd = "tar xpvf %s/rootfs.tar -C %s" % (cr_workdir, new_rootfs_dir) > + rc, out = exec_native_cmd(pseudox + tarx_cmd, native_sysroot) > + > + rm_cmd = "rm %s/rootfs.tar" % cr_workdir > + rc, out = exec_cmd(rm_cmd, True) > + > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > + > + return new_rootfs_dir > + > +class UBootPlugin(SourcePlugin): > + name = 'uboot' > + > + @classmethod > + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, krootfs_dir, > + bootimg_dir, kernel_dir, native_sysroot): > + """ > + Called before all partitions have been prepared and assembled into a > + disk image. Intall packages based on wic configuration. > + """ > + Same here, this looks like general-purpose package installation code that shouldn't be part of a u-boot plugin. Also, is that comment correct? > + # set new rootfs_dir > + rootfs_dir = create_local_rootfs(part, creator, cr_workdir, krootfs_dir, native_sysroot) > + > + # wks file parsing > + packages = kickstart.get_packages(creator.ks) > + > + # wic.conf file parsing = found under 'creator' > + local_pkgs_path = creator._local_pkgs_path > + repourl = creator.repourl > + pkgmgr = creator.pkgmgr_name > + > + # install packages > + if packages and pkgmgr in ["opkg"]: > + if len(repourl) > 0 : > + part.install_pkgs_ipk(cr_workdir, oe_builddir, rootfs_dir, native_sysroot, > + packages, repourl) > + else: > + msger.error("No packages repository provided in wic.conf") > + > + @classmethod > + def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, bootimg_dir, > + kernel_dir, krootfs_dir, native_sysroot): > + """ > + Called to do the actual content population for a partition i.e. it > + 'prepares' the partition to be incorporated into the image. > + In this case, prepare content for legacy bios boot partition. > + """ > + if part.rootfs is None: > + if not 'ROOTFS_DIR' in krootfs_dir: > + msg = "Couldn't find --rootfs-dir, exiting" > + msger.error(msg) > + rootfs_dir = krootfs_dir['ROOTFS_DIR'] > + else: > + if part.rootfs in krootfs_dir: > + rootfs_dir = krootfs_dir[part.rootfs] > + elif os.path.isdir(part.rootfs): > + rootfs_dir = part.rootfs > + else: > + msg = "Couldn't find --rootfs-dir=%s connection" > + msg += " or it is not a valid path, exiting" > + msger.error(msg % part.rootfs) > + > + part.set_rootfs(rootfs_dir) > + > + # change partition label wich will reflect into the final rootfs image name > + part.label = "%s_%s" % (part.label, cr.name) > + > + defpath = os.environ['PATH'] > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > + > + part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, native_sysroot) > + part.prepare_for_uboot(cr.target_arch,cr_workdir, oe_builddir, rootfs_dir, native_sysroot) > + > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > diff --git a/scripts/lib/mic/utils/oe/package_manager.py b/scripts/lib/mic/utils/oe/package_manager.py > new file mode 100644 > index 0000000..92ce98e > --- /dev/null > +++ b/scripts/lib/mic/utils/oe/package_manager.py > @@ -0,0 +1,810 @@ > +# ex:ts=4:sw=4:sts=4:et > +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > +# > +# Copyright (c) 2014, Enea AB. > +# All rights reserved. > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License version 2 as > +# published by the Free Software Foundation. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License along > +# with this program; if not, write to the Free Software Foundation, Inc., > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > +# > +# DESCRIPTION > +# This implements the opkg package manager wrapper as a combination of > +# meta/lib/oe/package_manager.py and bitbake/lib/bb/utils.py files and > +# adaptation of those files to 'wic'. > +# > +# AUTHORS > +# Adrian Calianu <adrian.calianu (at] enea.com> > +# > +# This file incorporates work covered by the following copyright and > +# permission notice: > +# > +# meta/COPYING.GPLv2 (GPLv2) > +# meta/COPYING.MIT (MIT) > +# > +# Copyright (C) 2004 Michael Lauer > +# > +# Permission to use, copy, modify, and/or distribute this software > +# for any purpose with or without fee is hereby granted, provided > +# that the above copyright notice and this permission notice appear > +# in all copies. > +# > +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL > +# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED > +# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE > +# AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR > +# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS > +# OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, > +# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN > +# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. > + > + > +from abc import ABCMeta, abstractmethod > +import os > +import glob > +import subprocess > +import shutil > +import multiprocessing > +import re > +import errno > +import fcntl > + > +from mic.utils.oe.misc import * > +from mic import msger > + > +def mkdirhier(directory): > + """Create a directory like 'mkdir -p', but does not complain if > + directory already exists like os.makedirs > + """ > + > + try: > + os.makedirs(directory) > + except OSError as e: > + if e.errno != errno.EEXIST: > + raise e > + > +def remove(path, recurse=False): > + """Equivalent to rm -f or rm -rf""" > + if not path: > + return > + if recurse: > + # shutil.rmtree(name) would be ideal but its too slow > + subprocess.call(['rm', '-rf'] + glob.glob(path)) > + return > + for name in glob.glob(path): > + try: > + os.unlink(name) > + except OSError as exc: > + if exc.errno != errno.ENOENT: > + raise > + > +def lockfile(name, shared=False, retry=True): > + """ > + Use the file fn as a lock file, return when the lock has been acquired. > + Returns a variable to pass to unlockfile(). > + """ > + dirname = os.path.dirname(name) > + mkdirhier(dirname) > + > + if not os.access(dirname, os.W_OK): > + logger.error("Unable to acquire lock '%s', directory is not writable", > + name) > + sys.exit(1) > + > + op = fcntl.LOCK_EX > + if shared: > + op = fcntl.LOCK_SH > + if not retry: > + op = op | fcntl.LOCK_NB > + > + while True: > + # If we leave the lockfiles lying around there is no problem > + # but we should clean up after ourselves. This gives potential > + # for races though. To work around this, when we acquire the lock > + # we check the file we locked was still the lock file on disk. > + # by comparing inode numbers. If they don't match or the lockfile > + # no longer exists, we start again. > + > + # This implementation is unfair since the last person to request the > + # lock is the most likely to win it. > + > + try: > + lf = open(name, 'a+') > + fileno = lf.fileno() > + fcntl.flock(fileno, op) > + statinfo = os.fstat(fileno) > + if os.path.exists(lf.name): > + statinfo2 = os.stat(lf.name) > + if statinfo.st_ino == statinfo2.st_ino: > + return lf > + lf.close() > + except Exception: > + try: > + lf.close() > + except Exception: > + pass > + pass > + if not retry: > + return None > + > +def unlockfile(lf): > + """ > + Unlock a file locked using lockfile() > + """ > + try: > + # If we had a shared lock, we need to promote to exclusive before > + # removing the lockfile. Attempt this, ignore failures. > + fcntl.flock(lf.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB) > + os.unlink(lf.name) > + except (IOError, OSError): > + pass > + fcntl.flock(lf.fileno(), fcntl.LOCK_UN) > + lf.close() > + > +def which(path, item, direction = 0, history = False): > + """ > + Locate a file in a PATH > + """ > + > + hist = [] > + paths = (path or "").split(':') > + if direction != 0: > + paths.reverse() > + > + for p in paths: > + next = os.path.join(p, item) > + hist.append(next) > + if os.path.exists(next): > + if not os.path.isabs(next): > + next = os.path.abspath(next) > + if history: > + return next, hist > + return next > + > + if history: > + return "", hist > + return "" > + > + > + > +# this can be used by all PM backends to create the index files in parallel > +def wic_create_index(arg): > + index_cmd = arg > + > + try: > + msger.info("Executing '%s' ..." % index_cmd) > + subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, shell=True) > + except subprocess.CalledProcessError as e: > + return("Index creation command '%s' failed with return code %d:\n%s" % > + (e.cmd, e.returncode, e.output)) > + > + return None > + > + > +class WicIndexer(object): > + __metaclass__ = ABCMeta > + > + def __init__(self, d, deploy_dir): > + self.d = d > + self.deploy_dir = deploy_dir > + > + @abstractmethod > + def write_index(self): > + pass > + > +class WicOpkgIndexer(WicIndexer): > + def write_index(self): > + arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS", > + "SDK_PACKAGE_ARCHS", > + "MULTILIB_ARCHS"] > + > + opkg_index_cmd = which(os.getenv('PATH'), "opkg-make-index") > + > + if not os.path.exists(os.path.join(self.deploy_dir, "Packages")): > + open(os.path.join(self.deploy_dir, "Packages"), "w").close() > + > + index_cmds = [] > + for arch_var in arch_vars: > + if self.d.has_key(arch_var): > + archs = self.d[arch_var] > + else: > + archs = None > + > + if archs is None: > + continue > + > + for arch in archs.split(): > + pkgs_dir = os.path.join(self.deploy_dir, arch) > + pkgs_file = os.path.join(pkgs_dir, "Packages") > + > + if not os.path.isdir(pkgs_dir): > + continue > + > + if not os.path.exists(pkgs_file): > + open(pkgs_file, "w").close() > + > + index_cmds.append('%s -r %s -p %s -m %s' % > + (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir)) > + > + if len(index_cmds) == 0: > + msger.info("There are no packages in %s!" % self.deploy_dir) > + return > + > + nproc = multiprocessing.cpu_count() > + pool = multiprocessing.Pool(nproc) > + results = list(pool.imap(wic_create_index, index_cmds)) > + pool.close() > + pool.join() > + > + for result in results: > + if result is not None: > + return(result) > + > +class WicPkgsList(object): > + __metaclass__ = ABCMeta > + > + def __init__(self, d, rootfs_dir): > + self.d = d > + self.rootfs_dir = rootfs_dir > + > + @abstractmethod > + def list(self, format=None): > + pass > + > + > +class WicOpkgPkgsList(WicPkgsList): > + def __init__(self, d, rootfs_dir, config_file): > + super(WicOpkgPkgsList, self).__init__(d, rootfs_dir) > + > + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") > + self.opkg_args = "-f %s -o %s " % (config_file, rootfs_dir) > + if self.d.has_key("OPKG_ARGS"): > + self.opkg_args += self.d["OPKG_ARGS"] > + > + def list(self, format=None): > + opkg_query_cmd = which(os.getenv('PATH'), "opkg-query-helper.py") > + > + if format == "arch": > + cmd = "%s %s status | %s -a" % \ > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > + elif format == "file": > + cmd = "%s %s status | %s -f" % \ > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > + elif format == "ver": > + cmd = "%s %s status | %s -v" % \ > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > + elif format == "deps": > + cmd = "%s %s status | %s" % \ > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > + else: > + cmd = "%s %s list_installed | cut -d' ' -f1" % \ > + (self.opkg_cmd, self.opkg_args) > + > + try: > + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip() > + except subprocess.CalledProcessError as e: > + msger.error("Cannot get the installed packages list. Command '%s' " > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > + > + if output and format == "file": > + tmp_output = "" > + for line in output.split('\n'): > + pkg, pkg_file, pkg_arch = line.split() > + full_path = os.path.join(self.rootfs_dir, pkg_arch, pkg_file) > + if os.path.exists(full_path): > + tmp_output += "%s %s %s\n" % (pkg, full_path, pkg_arch) > + else: > + tmp_output += "%s %s %s\n" % (pkg, pkg_file, pkg_arch) > + > + output = tmp_output > + > + return output > + > + > +class WicPackageManager(object): > + """ > + This is an abstract class. Do not instantiate this directly. > + """ > + __metaclass__ = ABCMeta > + > + def __init__(self, d, pseudo, native_sysroot): > + self.d = d > + self.deploy_dir = None > + self.deploy_lock = None > + if self.d.has_key('PACKAGE_FEED_URIS'): > + self.feed_uris = self.d['PACKAGE_FEED_URIS'] > + else: > + self.feed_uris = "" > + self.pseudo = pseudo > + self.native_sysroot = native_sysroot > + > + """ > + Update the package manager package database. > + """ > + @abstractmethod > + def update(self): > + pass > + > + """ > + Install a list of packages. 'pkgs' is a list object. If 'attempt_only' is > + True, installation failures are ignored. > + """ > + @abstractmethod > + def install(self, pkgs, attempt_only=False): > + pass > + > + """ > + Remove a list of packages. 'pkgs' is a list object. If 'with_dependencies' > + is False, the any dependencies are left in place. > + """ > + @abstractmethod > + def remove(self, pkgs, with_dependencies=True): > + pass > + > + """ > + This function creates the index files > + """ > + @abstractmethod > + def write_index(self): > + pass > + > + @abstractmethod > + def remove_packaging_data(self): > + pass > + > + @abstractmethod > + def list_installed(self, format=None): > + pass > + > + @abstractmethod > + def insert_feeds_uris(self): > + pass > + > + """ > + Install complementary packages based upon the list of currently installed > + packages e.g. locales, *-dev, *-dbg, etc. This will only attempt to install > + these packages, if they don't exist then no error will occur. Note: every > + backend needs to call this function explicitly after the normal package > + installation > + """ > + def install_complementary(self, globs=None): > + # we need to write the list of installed packages to a file because the > + # oe-pkgdata-util reads it from a file > + if self.d.has_key('WORKDIR'): > + installed_pkgs_file = os.path.join(self.d['WORKDIR'], > + "installed_pkgs.txt") > + else: > + msger.error("No WORKDIR provided!") > + > + with open(installed_pkgs_file, "w+") as installed_pkgs: > + installed_pkgs.write(self.list_installed("arch")) > + > + if globs is None: > + if self.d.has_key('IMAGE_INSTALL_COMPLEMENTARY'): > + globs = self.d['IMAGE_INSTALL_COMPLEMENTARY'] > + split_linguas = set() > + > + if self.d.has_key('IMAGE_LINGUAS'): > + for translation in self.d['IMAGE_LINGUAS'].split(): > + split_linguas.add(translation) > + split_linguas.add(translation.split('-')[0]) > + > + split_linguas = sorted(split_linguas) > + > + for lang in split_linguas: > + globs += " *-locale-%s" % lang > + > + if globs is None: > + return > + > + if not self.d.has_key('PKGDATA_DIR'): > + msger.error("No PKGDATA_DIR provided!") > + > + cmd = [which(os.getenv('PATH'), "oe-pkgdata-util"), > + "glob", self.d['PKGDATA_DIR'], installed_pkgs_file, > + globs] > + > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > + if rc != 0: > + msger.error("Could not compute complementary packages list. Command " > + "'%s' returned %d" % > + (' '.join(cmd), rc)) > + > + self.install(out.split(), attempt_only=True) > + > + > + def deploy_dir_lock(self): > + if self.deploy_dir is None: > + raise RuntimeError("deploy_dir is not set!") > + > + lock_file_name = os.path.join(self.deploy_dir, "deploy.lock") > + > + self.deploy_lock = lockfile(lock_file_name) > + > + def deploy_dir_unlock(self): > + if self.deploy_lock is None: > + return > + > + unlockfile(self.deploy_lock) > + > + self.deploy_lock = None > + > + > +class WicOpkgPM(WicPackageManager): > + def __init__(self, d, target_rootfs, config_file, archs, pseudo, native_sysroot, task_name='target'): > + super(WicOpkgPM, self).__init__(d, pseudo, native_sysroot) > + > + self.target_rootfs = target_rootfs > + self.config_file = config_file > + self.pkg_archs = archs > + self.task_name = task_name > + > + if self.d.has_key("DEPLOY_DIR_IPK"): > + self.deploy_dir = self.d["DEPLOY_DIR_IPK"] > + > + self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock") > + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") > + self.opkg_args = "-f %s -o %s " % (self.config_file, target_rootfs) > + if self.d.has_key("OPKG_ARGS"): > + self.opkg_args += self.d["OPKG_ARGS"] > + > + if self.d.has_key('OPKGLIBDIR'): > + opkg_lib_dir = self.d['OPKGLIBDIR'] > + else: > + opkg_lib_dir = "" > + > + if opkg_lib_dir[0] == "/": > + opkg_lib_dir = opkg_lib_dir[1:] > + > + self.opkg_dir = os.path.join(target_rootfs, opkg_lib_dir, "opkg") > + > + mkdirhier(self.opkg_dir) > + > + if self.d.has_key("TMPDIR"): > + tmp_dir = self.d["TMPDIR"] > + else: > + tmp_dir = "" > + > + self.saved_opkg_dir = '%s/saved/%s' % (tmp_dir, self.task_name) > + if not os.path.exists('%s/saved' % tmp_dir): > + mkdirhier('%s/saved' % tmp_dir) > + > + if self.d.has_key('BUILD_IMAGES_FROM_FEEDS') and self.d['BUILD_IMAGES_FROM_FEEDS'] != "1": > + self._create_config() > + else: > + self._create_custom_config() > + > + self.indexer = WicOpkgIndexer(self.d, self.deploy_dir) > + > + """ > + This function will change a package's status in /var/lib/opkg/status file. > + If 'packages' is None then the new_status will be applied to all > + packages > + """ > + def mark_packages(self, status_tag, packages=None): > + status_file = os.path.join(self.opkg_dir, "status") > + > + with open(status_file, "r") as sf: > + with open(status_file + ".tmp", "w+") as tmp_sf: > + if packages is None: > + tmp_sf.write(re.sub(r"Package: (.*?)\n((?:[^\n]+\n)*?)Status: (.*)(?:unpacked|installed)", > + r"Package: \1\n\2Status: \3%s" % status_tag, > + sf.read())) > + else: > + if type(packages).__name__ != "list": > + raise TypeError("'packages' should be a list object") > + > + status = sf.read() > + for pkg in packages: > + status = re.sub(r"Package: %s\n((?:[^\n]+\n)*?)Status: (.*)(?:unpacked|installed)" % pkg, > + r"Package: %s\n\1Status: \2%s" % (pkg, status_tag), > + status) > + > + tmp_sf.write(status) > + > + os.rename(status_file + ".tmp", status_file) > + > + def _create_custom_config(self): > + msger.info("Building from feeds activated!") > + > + with open(self.config_file, "w+") as config_file: > + priority = 1 > + for arch in self.pkg_archs.split(): > + config_file.write("arch %s %d\n" % (arch, priority)) > + priority += 5 > + > + if self.d.has_key('IPK_FEED_URIS'): > + ipk_feed_uris = self.d['IPK_FEED_URIS'] > + else: > + ipk_feed_uris = "" > + > + for line in ipk_feed_uris.split(): > + feed_match = re.match("^[ \t]*(.*)##([^ \t]*)[ \t]*$", line) > + > + if feed_match is not None: > + feed_name = feed_match.group(1) > + feed_uri = feed_match.group(2) > + > + msger.info("Add %s feed with URL %s" % (feed_name, feed_uri)) > + > + config_file.write("src/gz %s %s\n" % (feed_name, feed_uri)) > + > + """ > + Allow to use package deploy directory contents as quick devel-testing > + feed. This creates individual feed configs for each arch subdir of those > + specified as compatible for the current machine. > + NOTE: Development-helper feature, NOT a full-fledged feed. > + """ > + if self.d.has_key('FEED_DEPLOYDIR_BASE_URI'): > + feed_deploydir_base_dir = self.d['FEED_DEPLOYDIR_BASE_URI'] > + else: > + feed_deploydir_base_dir = "" > + > + if feed_deploydir_base_dir != "": > + for arch in self.pkg_archs.split(): > + if self.d.has_key("sysconfdir"): > + sysconfdir = self.d["sysconfdir"] > + else: > + sysconfdir = None > + > + cfg_file_name = os.path.join(self.target_rootfs, > + sysconfdir, > + "opkg", > + "local-%s-feed.conf" % arch) > + > + with open(cfg_file_name, "w+") as cfg_file: > + cfg_file.write("src/gz local-%s %s/%s" % > + arch, > + feed_deploydir_base_dir, > + arch) > + > + def _create_config(self): > + with open(self.config_file, "w+") as config_file: > + priority = 1 > + for arch in self.pkg_archs.split(): > + config_file.write("arch %s %d\n" % (arch, priority)) > + priority += 5 > + > + config_file.write("src oe file:%s\n" % self.deploy_dir) > + > + for arch in self.pkg_archs.split(): > + pkgs_dir = os.path.join(self.deploy_dir, arch) > + if os.path.isdir(pkgs_dir): > + config_file.write("src oe-%s file:%s\n" % > + (arch, pkgs_dir)) > + > + def insert_feeds_uris(self): > + if self.feed_uris == "": > + return > + > + rootfs_config = os.path.join('%s/etc/opkg/base-feeds.conf' > + % self.target_rootfs) > + > + with open(rootfs_config, "w+") as config_file: > + uri_iterator = 0 > + for uri in self.feed_uris.split(): > + config_file.write("src/gz url-%d %s/ipk\n" % > + (uri_iterator, uri)) > + > + for arch in self.pkg_archs.split(): > + if not os.path.exists(os.path.join(self.deploy_dir, arch)): > + continue > + msger.info('Note: adding opkg channel url-%s-%d (%s)' % > + (arch, uri_iterator, uri)) > + > + config_file.write("src/gz uri-%s-%d %s/ipk/%s\n" % > + (arch, uri_iterator, uri, arch)) > + uri_iterator += 1 > + > + def update(self): > + self.deploy_dir_lock() > + > + cmd = "%s %s update" % (self.opkg_cmd, self.opkg_args) > + > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > + if rc != 0: > + self.deploy_dir_unlock() > + msger.error("Unable to update the package index files. Command '%s' " > + "returned %d" % (cmd, rc)) > + > + self.deploy_dir_unlock() > + > + def install(self, pkgs, attempt_only=False): > + if attempt_only and len(pkgs) == 0: > + return > + > + cmd = "%s %s install %s" % (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) > + > + os.environ['D'] = self.target_rootfs > + os.environ['OFFLINE_ROOT'] = self.target_rootfs > + os.environ['IPKG_OFFLINE_ROOT'] = self.target_rootfs > + os.environ['OPKG_OFFLINE_ROOT'] = self.target_rootfs > + if self.d.has_key('WORKDIR'): > + os.environ['INTERCEPT_DIR'] = os.path.join(self.d['WORKDIR'], > + "intercept_scripts") > + else: > + os.environ['INTERCEPT_DIR'] = "." > + msger.warning("No WORKDIR provided!") > + > + if self.d.has_key('STAGING_DIR_NATIVE'): > + os.environ['NATIVE_ROOT'] = self.d['STAGING_DIR_NATIVE'] > + else: > + msger.error("No STAGING_DIR_NATIVE provided!") > + > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > + if rc != 0: > + msger.error("Unable to install packages. " > + "Command '%s' returned %d" % (cmd, rc)) > + > + > + def remove(self, pkgs, with_dependencies=True): > + if with_dependencies: > + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \ > + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) > + else: > + cmd = "%s %s --force-depends remove %s" % \ > + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) > + > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > + if rc != 0: > + msger.error("Unable to remove packages. Command '%s' " > + "returned %d" % (cmd, rc)) > + > + > + def write_index(self): > + self.deploy_dir_lock() > + > + result = self.indexer.write_index() > + > + self.deploy_dir_unlock() > + > + if result is not None: > + msger.error(result) > + > + def remove_packaging_data(self): > + remove(self.opkg_dir, True) > + # create the directory back, it's needed by PM lock > + mkdirhier(self.opkg_dir) > + > + def list_installed(self, format=None): > + return WicOpkgPkgsList(self.d, self.target_rootfs, self.config_file).list(format) > + > + def handle_bad_recommendations(self): > + if self.d.has_key("BAD_RECOMMENDATIONS"): > + bad_recommendations = self.d["BAD_RECOMMENDATIONS"] > + else: > + bad_recommendations = "" > + > + if bad_recommendations.strip() == "": > + return > + > + status_file = os.path.join(self.opkg_dir, "status") > + > + # If status file existed, it means the bad recommendations has already > + # been handled > + if os.path.exists(status_file): > + return > + > + cmd = "%s %s info " % (self.opkg_cmd, self.opkg_args) > + > + with open(status_file, "w+") as status: > + for pkg in bad_recommendations.split(): > + pkg_info = cmd + pkg > + > + try: > + output = subprocess.check_output(pkg_info.split(), stderr=subprocess.STDOUT).strip() > + except subprocess.CalledProcessError as e: > + msger.error("Cannot get package info. Command '%s' " > + "returned %d:\n%s" % (pkg_info, e.returncode, e.output)) > + > + if output == "": > + msger.info("Ignored bad recommendation: '%s' is " > + "not a package" % pkg) > + continue > + > + for line in output.split('\n'): > + if line.startswith("Status:"): > + status.write("Status: deinstall hold not-installed\n") > + else: > + status.write(line + "\n") > + > + ''' > + The following function dummy installs pkgs and returns the log of output. > + ''' > + def dummy_install(self, pkgs): > + if len(pkgs) == 0: > + return > + > + # Create an temp dir as opkg root for dummy installation > + if self.d.has_key("TMPDIR"): > + tmp_dir = self.d["TMPDIR"] > + else: > + tmp_dir = "." > + msger.warning("No TMPDIR provided!") > + > + temp_rootfs = '%s/opkg' % tmp_dir > + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') > + mkdirhier(temp_opkg_dir) > + > + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) > + if self.d.has_key("OPKG_ARGS"): > + opkg_args += self.d["OPKG_ARGS"] > + > + cmd = "%s %s update" % (self.opkg_cmd, opkg_args) > + try: > + subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) > + except subprocess.CalledProcessError as e: > + msger.error("Unable to update. Command '%s' " > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > + > + # Dummy installation > + cmd = "%s %s --noaction install %s " % (self.opkg_cmd, > + opkg_args, > + ' '.join(pkgs)) > + try: > + output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True) > + except subprocess.CalledProcessError as e: > + msger.error("Unable to dummy install packages. Command '%s' " > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > + > + remove(temp_rootfs, True) > + > + return output > + > + def backup_packaging_data(self): > + # Save the opkglib for increment ipk image generation > + if os.path.exists(self.saved_opkg_dir): > + remove(self.saved_opkg_dir, True) > + shutil.copytree(self.opkg_dir, > + self.saved_opkg_dir, > + symlinks=True) > + > + def recover_packaging_data(self): > + # Move the opkglib back > + if os.path.exists(self.saved_opkg_dir): > + if os.path.exists(self.opkg_dir): > + remove(self.opkg_dir, True) > + > + msger.info('Recover packaging data') > + shutil.copytree(self.saved_opkg_dir, > + self.opkg_dir, > + symlinks=True) > + > + > +def wic_generate_index_files(d): > + if d.has_key('PACKAGE_CLASSES'): > + classes = d['PACKAGE_CLASSES'].replace("package_", "").split() > + else: > + classes = "" > + msger.warning("No PACKAGE_CLASSES provided!") > + > + if d.has_key('DEPLOY_DIR_IPK'): > + deploy_dir_ipk = d['DEPLOY_DIR_IPK'] > + else: > + deploy_dir_ipk = None > + msger.warning("No DEPLOY_DIR_IPK provided!") > + > + indexer_map = { > + "ipk": (WicOpkgIndexer, deploy_dir_ipk) > + } > + > + result = None > + > + for pkg_class in classes: > + if not pkg_class in indexer_map: > + continue > + > + if os.path.exists(indexer_map[pkg_class][1]): > + result = indexer_map[pkg_class][0](d, indexer_map[pkg_class][1]).write_index() > + > + if result is not None: > + msger.error(result) > diff --git a/scripts/wic b/scripts/wic > index 2d3fd09..b9c8756 100755 > --- a/scripts/wic > +++ b/scripts/wic > @@ -99,6 +99,10 @@ def wic_create_subcommand(args, usage_str): > > (options, args) = parser.parse_args(args) > > + if options.debug: > + loglevel = logging.DEBUG > + start_logging(loglevel) > + > if len(args) != 1: > logging.error("Wrong number of arguments, exiting\n") > parser.print_help() > @@ -107,9 +111,11 @@ def wic_create_subcommand(args, usage_str): > if not options.image_name and not (options.rootfs_dir and > options.bootimg_dir and > options.kernel_dir and > + options.native_sysroot or > options.native_sysroot): > print "Build artifacts not completely specified, exiting." > - print " (Use 'wic -e' or 'wic -r -b -k -n' to specify artifacts)" > + print " (Use 'wic -e' or 'wic -r -b -k -n' or 'wic -r -n' to specify artifacts)" > + print options > sys.exit(1) > > if not options.image_name: > @@ -125,13 +131,16 @@ def wic_create_subcommand(args, usage_str): > > print "Creating image(s)...\n" > > - bitbake_env_lines = find_bitbake_env_lines(options.image_name) > - if not bitbake_env_lines: > - print "Couldn't get bitbake environment, exiting." > - sys.exit(1) > - set_bitbake_env_lines(bitbake_env_lines) > + # If '-e' option is used the values are extracted from bitbake env. > + if options.image_name: > + bitbake_env_lines = find_bitbake_env_lines(options.image_name) > + if not bitbake_env_lines: > + print "Couldn't get bitbake environment, exiting." > + sys.exit(1) > + set_bitbake_env_lines(bitbake_env_lines) > > bootimg_dir = staging_data_dir = hdddir = "" > + rootfs_dir = native_sysroot = kernel_dir = image_output_dir = "" > > if options.image_name: > (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot) = \ > @@ -140,34 +149,65 @@ def wic_create_subcommand(args, usage_str): > wks_file = args[0] > > if not wks_file.endswith(".wks"): > + # Return full path of the .wks file > wks_file = find_canned_image(scripts_path, wks_file) > if not wks_file: > - print "No image named %s found, exiting. (Use 'wic list images' to list available images, or specify a fully-qualified OE kickstart (.wks) filename)\n" % wks_file > + print "No image named %s found, exiting.\n" % wks_file > + print "(Use 'wic list images' to list available images, or specify a fully-qualified OE kickstart (.wks) filename)\n" > sys.exit(1) > > - image_output_dir = "" > if options.outdir: > image_output_dir = options.outdir > > - if not options.image_name: > - rootfs_dir = '' > - if 'ROOTFS_DIR' in options.rootfs_dir: > - rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > - bootimg_dir = options.bootimg_dir > - kernel_dir = options.kernel_dir > + if options.native_sysroot: > native_sysroot = options.native_sysroot > - if rootfs_dir and not os.path.isdir(rootfs_dir): > - print "--roofs-dir (-r) not found, exiting\n" > - sys.exit(1) > - if not os.path.isdir(bootimg_dir): > - print "--bootimg-dir (-b) not found, exiting\n" > - sys.exit(1) > - if not os.path.isdir(kernel_dir): > - print "--kernel-dir (-k) not found, exiting\n" > - sys.exit(1) > + print "Using native_sysroot from user command: %s" % native_sysroot > + > if not os.path.isdir(native_sysroot): > - print "--native-sysroot (-n) not found, exiting\n" > + print "--native-sysroot (-n) not found, exiting" > sys.exit(1) > + > + native_sysroot = os.path.abspath(native_sysroot) > + > + if not options.image_name: > + if (options.bootimg_dir and options.kernel_dir and > + options.rootfs_dir and options.native_sysroot): > + rootfs_dir = '' > + if 'ROOTFS_DIR' in options.rootfs_dir: > + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > + bootimg_dir = options.bootimg_dir > + kernel_dir = options.kernel_dir > + native_sysroot = options.native_sysroot > + > + if rootfs_dir and not os.path.isdir(rootfs_dir): > + print "--roofs-dir (-r) not found, exiting\n" > + sys.exit(1) > + if not os.path.isdir(bootimg_dir): > + print "--bootimg-dir (-b) not found, exiting\n" > + sys.exit(1) > + if not os.path.isdir(kernel_dir): > + print "--kernel-dir (-k) not found, exiting\n" > + sys.exit(1) > + if not os.path.isdir(native_sysroot): > + print "--native-sysroot (-n) not found, exiting\n" > + sys.exit(1) > + else: > + print 'Build image from rootfs and a package list using native rootfs\n' > + if options.rootfs_dir and 'ROOTFS_DIR' in options.rootfs_dir: > + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > + elif options.rootfs_dir: > + rootfs_dir = options.rootfs_dir > + else: > + rootfs_dir = "" > + > + native_sysroot = options.native_sysroot > + > + if rootfs_dir and not os.path.isdir(rootfs_dir): > + print "--roofs-dir (-r) not found, exiting\n" > + sys.exit(1) > + if not os.path.isdir(native_sysroot): > + print "--native-sysroot (-n) not found, exiting\n" > + sys.exit(1) > else: > not_found = not_found_dir = "" > if not os.path.isdir(rootfs_dir): > -- > 1.7.10.4 > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv2 1/2] Add wic support to generate rootfs image for uboot 2014-06-13 20:18 ` Tom Zanussi @ 2014-06-16 13:18 ` Adrian Calianu 2014-06-17 9:35 ` Anders Darander 2014-06-20 14:46 ` Tom Zanussi 0 siblings, 2 replies; 11+ messages in thread From: Adrian Calianu @ 2014-06-16 13:18 UTC (permalink / raw) To: Tom Zanussi, Adrian Calianu Cc: Otavio Salvador, openembedded-core@lists.openembedded.org Hi Tom, First of all, thanks for your review comments! I'm glad that community has interest for this features. I will answer to your comments in-line below... > -----Original Message----- > From: openembedded-core-bounces@lists.openembedded.org > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf > Of Tom Zanussi > Sent: Friday, June 13, 2014 11:19 PM > To: y@enea.se > Cc: openembedded-core@lists.openembedded.org; Otavio Salvador > Subject: Re: [OE-core] [PATCHv2 1/2] Add wic support to generate rootfs > image for uboot > > Hi, > > This looks like some nice new functionality, but I have some suggestions > regarding the implementation and the posting itself... > > First, this seems to have been sent by a 'y' at enea, but they're > apparently written by adrian.calianu at enea - why can't adrian.calianu > send them himself? (makes me wonder about the authorship, etc). [Adrian Calianu] It was an issue with local git server since the patch was send directly from git but nothing to worry about, problem was fixed. > > Also, there seems to be just one big patch, but the subject says > '[PATCHv2 1/2] Add wic support to generate rootfs image for uboot' - am > I missing 2/2? [Adrian Calianu] Actually there were two versions of patch and for each patch a cover letter was also generated which lead to those numbers. > > As for the patch itself, it's huge and needs to be broken up into > smaller pieces. In fact, it seems to be a combination of two completely > separate things at least - a new mechanism for creating a rootfs from > packages, and a new plugin for creating uboot partitions (which just > happens to use the output from the new packaging step. So at minimum I > think you need to break it up along the lines of that functionality, and > it would be good to break it up even further e.g. 'Add a new do_pkg > interface hook', 'Add an implementation of do_pkg for ipk', 'Add a new > source plugin for uboot partitions', 'Add a new canned wks for a default > uboot image', etc. The more you can break it up into logical chunks > without going overboard, the easier it is to review and apply/revert. [Adrian Calianu] I agree that there are two main features but the reason of not break it in small pieces now was to provide a fully functional patch for community to test it and be able to provide a feedback for this patch, since those features cannot be tested independently now. So, is it acceptable to provide patches that cannot be tested? > > Finally, something like this really needs to be documented in the tool - > please add the relevant interface documentation to the help system. [Adrian Calianu] Ok, I will investigate and add more info to help system. I saw that the Yocto manual included wic doc only in 1.5.2 version(http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html). Why that info is not found into the latest Yocto manual? It is anything planned about that? > > For the 'packaging' part of this patch, the big problem I see is that > the package_manager seems to be basically a copy of the oe package > manager and more (which you mention in the comments) - I don't think it > makes much sense to do that - wouldn't it be better to patch the oe > package_manager code so that it could be reused by this tool? [Adrian Calianu] This is a subject of discussion I think. Yes, first time I just used the modules from bitbake and oe and worked perfectly but this means to link wic tool by bitbake and all his specific features like BB_variables, dataSmart,... Is this acceptable? Shouldn't wic tool be as independent as possible? What is the plan for wic tool in future? I see that the wic tool came with backend plugins (yum package manager). So depending on the plans for wic tool in future we may move this new package manager into backend plugin folder or use the one from bitbake also for rpm packages and remove existing one. I see it as an open subject. > > For the 'uboot' part, is the 'uboot' plugin implementing common > functionality that all uboot images need, or is it for a specific use > case (if so, it shouldn't be called 'uboot' but something more > specific)? Otavio seemed to suggest that in general it would need to be > modified for specific cases. I'm not sure about the best way to do that > - we don't have many good examples yet. Can that sort of thing be > parameterized/subclassed, or would it require a per-image plugin, which > would be good to avoid. Not something you have to solve now, if the > 'uboot' plugin is useful on its own and maybe subclassed/added onto by > future plugins. It would be nice to have some specific and succinct > examples to help think through what might be needed here. [Adrian Calianu] I was thinking it as a general uboot plugin and in future parameterized it like other existing plugins. We have plans to extend it for SD/MMC cards. > > Some more specific comments in-line below... > > On Thu, 2014-06-12 at 10:43 +0200, y@enea.se wrote: > > From: Adrian Calianu <adrian.calianu@enea.com> > > > > Add support in wic to build ramdisk uboot images from list of packages or > > from an existing rootfs folder. > > > > Providing the list of packages starting with XXX-core-boot package(into wks > file) > > will generate an rootfs image for uboot out of bitbake. > > > > There is also possible to combine an existing/application rootfs with a list > > of packages that can be installed but the user must take care to provide > > a basic core boot rootfs or XXX-core-boot package and reporul(wic.conf). > > > > Is this implemented in the current patch? [Adrian Calianu] Yes. This can be achieved by providing a valid basic core boot package(into wic.conf) and user rootfs(wic create -r "path_to_user_rootfs" -n "path_to_native_rootfs" ). > > > Some prerequisites are required for this new build setup: > > ../poky/scripts/lib/image/config/wic.conf > > [create] > > arch=target_arch (example: powerpc, arm, x86) > > pkgmgr=opkg > > repourl=http://example.distro/p2020rdb/ipk/all > http://example.distro/p2020rdb/ipk/p2020rdb > http://example.distro/p2020rdb/ipk/ppce500v2 > > > > 1) Build an rootfs image from an existing bitbake build: > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core- > image-minimal -o .../output > > > > 2) Build an rootfs image from an existing rootfs and native_sysroot: > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r > .../path_to_rootfs -n .../path_to_native_sysroot -o .../output > > > > 3) Build an rootfs image only from a package list (on wks file): > > > > Add the package list to be installed as rootfs on > ../poky/scripts/lib/image/canned-wks/uboot.wks: > > %packages > > packagegroup-core-boot > > pramfs-init > > run-postinsts > > packagegroup-core-ssh-dropbear > > %end > > > > Generate rootfs image: > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n > .../path_to_native_sysroot -o .../output > > > > Signed-off-by: Adrian Calianu <adrian.calianu@enea.com> > > --- > > scripts/lib/image/canned-wks/uboot.wks | 17 + > > scripts/lib/image/config/wic.conf | 4 + > > scripts/lib/mic/imager/direct.py | 4 + > > .../lib/mic/kickstart/custom_commands/partition.py | 128 ++++ > > scripts/lib/mic/pluginbase.py | 9 + > > scripts/lib/mic/plugins/source/uboot.py | 173 +++++ > > scripts/lib/mic/utils/oe/package_manager.py | 810 > ++++++++++++++++++++ > > scripts/wic | 88 ++- > > 10 files changed, 1219 insertions(+), 32 deletions(-) > > create mode 100644 scripts/lib/image/canned-wks/uboot.wks > > create mode 100644 scripts/lib/mic/plugins/source/uboot.py > > create mode 100644 scripts/lib/mic/utils/oe/package_manager.py > > > > diff --git a/scripts/lib/image/canned-wks/uboot.wks > b/scripts/lib/image/canned-wks/uboot.wks > > new file mode 100644 > > index 0000000..7de0572 > > --- /dev/null > > +++ b/scripts/lib/image/canned-wks/uboot.wks > > @@ -0,0 +1,17 @@ > > +# short-description: . Create a ramdisk image for U-Boot > > +# long-description: Creates a ramdisk image for U-Boot that user > > +# can directly load it into ram through tftp > > +# > > +# part - is a wic command that drive the process of generating a valid file > system > > +# - --source=uboot : wic plugin that generates a ramdisk image for U- > Boot > > +# - --fstype=ext2 : file system type( ext2 / ext3 / ext 4) > > +# > > +# %packages %end - option to provide a list of packages that will be > installed > > +# into rootfs. All packages dependencies will be installed by > > +# package manager(default opkg). > > + > > + > > +part / --source=uboot --fstype=ext2 --label imageName --align 1024 > > + > > +%packages > > +%end > > diff --git a/scripts/lib/image/config/wic.conf > b/scripts/lib/image/config/wic.conf > > index e96d6ae..2a2750b 100644 > > --- a/scripts/lib/image/config/wic.conf > > +++ b/scripts/lib/image/config/wic.conf > > @@ -4,4 +4,8 @@ distro_name = OpenEmbedded > > > > [create] > > ; settings for create subcommand > > +; repourl=http://linux.com/ipk/all http://linux.com/ipk/target > http://linux.com/ipk/arch > > +arch=powerpc > > +pkgmgr=opkg > > runtime=native > > +install_pkgs=source > > I don't think you want these settings as defaults in wic.conf. Maybe in > the documentation? [Adrian Calianu] What documentation? Please can you shade some light here? Sometimes from user perspective would be nice to have a fully functional features without needed to read a manual to understand it but here is difficult since settings are distribution dependent. > > > diff --git a/scripts/lib/mic/imager/direct.py > b/scripts/lib/mic/imager/direct.py > > index 2cf4c8d..fef9d0e 100644 > > --- a/scripts/lib/mic/imager/direct.py > > +++ b/scripts/lib/mic/imager/direct.py > > @@ -262,6 +262,10 @@ class DirectImageCreator(BaseImageCreator): > > # when/if we need to actually do package selection we > > # should modify things to use those objects, but for now > > # we can avoid that. > > + > > + p.install_pkgs(self, self.workdir, self.oe_builddir, self.rootfs_dir, > > + self.bootimg_dir, self.kernel_dir, self.native_sysroot) > > + > > p.prepare(self, self.workdir, self.oe_builddir, self.rootfs_dir, > > self.bootimg_dir, self.kernel_dir, self.native_sysroot) > > > > diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py > b/scripts/lib/mic/kickstart/custom_commands/partition.py > > index 6b575c0..450d2d4 100644 > > --- a/scripts/lib/mic/kickstart/custom_commands/partition.py > > +++ b/scripts/lib/mic/kickstart/custom_commands/partition.py > > @@ -31,7 +31,11 @@ from mic.utils.oe.misc import * > > from mic.kickstart.custom_commands import * > > from mic.plugin import pluginmgr > > > > +import os > > +from mic.utils.oe.package_manager import * > > + > > partition_methods = { > > + "do_install_pkgs":None, > > "do_stage_partition":None, > > "do_prepare_partition":None, > > "do_configure_partition":None, > > @@ -115,6 +119,102 @@ class Wic_PartData(Mic_PartData): > > else: > > return 0 > > > > + def install_pkgs(self, creator, cr_workdir, oe_builddir, rootfs_dir, > > + bootimg_dir, kernel_dir, native_sysroot): > > + """ > > + Prepare content for individual partitions, installing packages. > > + """ > > + > > + if not self.source: > > + return > > + > > + self._source_methods = > pluginmgr.get_source_plugin_methods(self.source, partition_methods) > > + self._source_methods["do_install_pkgs"](self, creator, > > + cr_workdir, > > + oe_builddir, > > + rootfs_dir, > > + bootimg_dir, > > + kernel_dir, > > + native_sysroot) > > + > > + def install_pkgs_ipk(self, cr_workdir, oe_builddir, rootfs_dir, > > + native_sysroot, packages, repourl): > > + """ > > + Install packages specified into wks file using opkg package manager. > > + This method is dependend on bb module. > > + """ > > + > > + gVar = {} > > + gVar["DEPLOY_DIR_IPK"] = os.path.join(oe_builddir, > "tmp/deploy/ipk") > > + > > + # Run postinstall scripts even in offline mode > > + # Use the arch priority package rather than higher version one if more > than one candidate is found. > > + #d.setVar("OPKG_ARGS", "--force_postinstall --prefer-arch-to- > version") > > + gVar["OPKG_ARGS"] = "--force_postinstall" > > + > > + # OPKG path relative to /output_path > > + gVar["OPKGLIBDIR"] = "var/lib" > > + > > + source_url = repourl.split() > > + > > + # Generate feed uri's names, it doesn't seem to matter what name > they have > > + feed_uris = "" > > + cnt = 0 > > + archs = "" > > + for url in source_url: > > + feed_uris += "cl_def_feed%d##%s\n" % (cnt, url) > > + cnt += 1 > > + head, tail = os.path.split(url) > > + archs += " " + tail > > + > > + # IPK_FEED_URIS with special formating defines the URI's used as > source for packages > > + gVar['IPK_FEED_URIS'] = feed_uris > > + > > + gVar['BUILD_IMAGES_FROM_FEEDS'] = "1" > > + > > + # We need to provide sysroot for utilities > > + gVar['STAGING_DIR_NATIVE'] = native_sysroot > > + > > + # Set WORKDIR for output > > + gVar['WORKDIR'] = cr_workdir > > + > > + # Set TMPDIR for output > > + gVar['TMPDIR'] = os.path.join(cr_workdir, "tmp") > > + > > + if 'ROOTFS_DIR' in rootfs_dir: > > + target_dir = rootfs_dir['ROOTFS_DIR'] > > + elif os.path.isdir(rootfs_dir): > > + target_dir = rootfs_dir > > + else: > > + msg = "Couldn't find --rootfs-dir=%s connection" > > + msg += " or it is not a valid path, exiting" > > + msger.error(msg % rootfs_dir) > > + > > + # Need native sysroot /usr/bin/ for opkg-cl > > + # chnage PATH var to avoid issues with host tools > > + defpath = os.environ['PATH'] > > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > > + > > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % > target_dir > > + pseudo += "export PSEUDO_PASSWD=%s;" % target_dir > > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > > + pseudo += "%s/usr/bin/pseudo " % native_sysroot > > + > > + pm = WicOpkgPM(gVar, > > + target_dir, > > + 'opkg.conf', > > + archs, > > + pseudo, > > + native_sysroot) > > + > > + pm.update() > > + > > + pm.install(packages) > > + > > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > > + > > + > > def prepare(self, cr, cr_workdir, oe_builddir, rootfs_dir, bootimg_dir, > > kernel_dir, native_sysroot): > > """ > > @@ -225,6 +325,34 @@ class Wic_PartData(Mic_PartData): > > > > return 0 > > > > + def prepare_for_uboot(self, arch, cr_workdir, oe_builddir, rootfs_dir, > > + native_sysroot): > > + """ > > + Generates u-boot image from source_file( ext2/3/4 ) > > + > > + """ > > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % > rootfs_dir > > + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir > > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > > + pseudo += "%s/usr/bin/pseudo " % native_sysroot > > + > > + # 1) compress image > > + rootfs = self.source_file > > + rootfs_gzip = "%s.gz" % rootfs > > + gzip_cmd = "gzip -f -9 -c %s > %s" % (rootfs, rootfs_gzip) > > + rc, out = exec_native_cmd(pseudo + gzip_cmd, native_sysroot) > > + > > + # 2) image for U-Boot > > + rootfs_uboot = "%s.u-boot" % rootfs_gzip > > + mkimage_cmd = "mkimage -A %s -O linux -T ramdisk -C gzip -n %s -d > %s %s" % \ > > + (arch, self.label, rootfs_gzip, rootfs_uboot) > > + rc, out = exec_native_cmd(pseudo + mkimage_cmd, native_sysroot) > > + > > + msger.info("\n\n\tThe new U-Boot ramdisk image can be found > here:\n\t\t%s\n\n" % rootfs_uboot) > > + > > + return 0 > > + > > def prepare_rootfs_btrfs(self, cr_workdir, oe_builddir, rootfs_dir, > > native_sysroot, pseudo): > > """ > > diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py > > index 9cf4c62..881d996 100644 > > --- a/scripts/lib/mic/pluginbase.py > > +++ b/scripts/lib/mic/pluginbase.py > > @@ -89,6 +89,15 @@ class SourcePlugin(_Plugin): > > """ > > > > @classmethod > > + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, > rootfs_dir, > > + bootimg_dir, kernel_dir, native_sysroot): > > + """ > > + Called before partitions have been prepared and assembled into a > > + disk image. Install packages into rootfs > > + """ > > + msger.debug("SourcePlugin: do_install_pkgs: part %s" % part) > > + > > + @classmethod > > def do_install_disk(self, disk, disk_name, cr, workdir, oe_builddir, > > bootimg_dir, kernel_dir, native_sysroot): > > """ > > diff --git a/scripts/lib/mic/plugins/source/uboot.py > b/scripts/lib/mic/plugins/source/uboot.py > > new file mode 100644 > > index 0000000..57cb3cf > > --- /dev/null > > +++ b/scripts/lib/mic/plugins/source/uboot.py > > @@ -0,0 +1,173 @@ > > +# ex:ts=4:sw=4:sts=4:et > > +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > > +# > > +# Copyright (c) 2014, Enea AB. > > +# All rights reserved. > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License version 2 as > > +# published by the Free Software Foundation. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > along > > +# with this program; if not, write to the Free Software Foundation, Inc., > > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > > +# > > +# DESCRIPTION > > +# This implements the 'uboot' source plugin class for 'wic' > > +# > > +# AUTHORS > > +# Adrian Calianu <adrian.calianu (at] enea.com> > > +# > > + > > +import os > > +import shutil > > +import re > > +import tempfile > > + > > +from mic import kickstart, chroot, msger > > +from mic.utils import misc, fs_related, errors, runner, cmdln > > +from mic.conf import configmgr > > +from mic.plugin import pluginmgr > > +from mic.utils.partitionedfs import PartitionedMount > > +import mic.imager.direct as direct > > +from mic.pluginbase import SourcePlugin > > +from mic.utils.oe.misc import * > > +from mic.imager.direct import DirectImageCreator > > + > > +def create_local_rootfs(part, creator, cr_workdir, krootfs_dir, > native_sysroot): > > + # In order to have a full control over rootfs we will make a local copy > under workdir > > + # and change rootfs_dir to new location. > > + # In this way we can install more than one ROOTFS_DIRs and/or use > > + # an empty rootfs to install packages, so a rootfs could be generated > only from pkgs > > + # TBD: create workdir/rootfs ; copy rootfs-> workdir/rootfs; set > rootfs=workdir/rootfs > > + > > Is this something that would be needed by any image creating a rootfs > from packages? If so, it should be part of the packaging functionality, > not the uboot implementation specifically. [Adrian Calianu] Yes, indeed it can be moved outside of uboot plugin. > > > + cr_workdir = os.path.abspath(cr_workdir) > > + new_rootfs_dir = "%s/rootfs_%s" % (cr_workdir, creator.name) > > + > > + rootfs_exists = 1 > > + if part.rootfs is None: > > + if not 'ROOTFS_DIR' in krootfs_dir: > > + msg = "Couldn't find --rootfs-dir, exiting, " > > + msger.info(msg) > > + rootfs_exists = 0 > > + rootfs_dir = krootfs_dir['ROOTFS_DIR'] > > + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir > > + else: > > + if part.rootfs in krootfs_dir: > > + rootfs_dir = krootfs_dir[part.rootfs] > > + creator.rootfs_dir[part.rootfs] = new_rootfs_dir > > + elif os.path.isdir(part.rootfs): > > + rootfs_dir = part.rootfs > > + part.rootfs = new_rootfs_dir > > + else: > > + msg = "Couldn't find --rootfs-dir=%s connection" > > + msg += " or it is not a valid path, exiting" > > + msger.info(msg % part.rootfs) > > + rootfs_exists = 0 > > + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir > > + > > + pseudox = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > > + pseudox += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % > new_rootfs_dir > > + pseudox += "export PSEUDO_PASSWD=%s;" % new_rootfs_dir > > + pseudox += "export PSEUDO_NOSYMLINKEXP=1;" > > + pseudox += "%s/usr/bin/pseudo " % native_sysroot > > + > > + mkdir_cmd = "mkdir %s" % (new_rootfs_dir) > > + # rc, out = exec_native_cmd(pseudox + mkdir_cmd, native_sysroot) > > + rc, out = exec_cmd(mkdir_cmd, True) > > + > > + if rootfs_exists == 1 and os.path.isdir(rootfs_dir): > > + defpath = os.environ['PATH'] > > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > > + > > + rootfs_dir = os.path.abspath(rootfs_dir) > > + > > + pseudoc = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > > + pseudoc += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % > rootfs_dir > > + pseudoc += "export PSEUDO_PASSWD=%s;" % rootfs_dir > > + pseudoc += "export PSEUDO_NOSYMLINKEXP=1;" > > + pseudoc += "%s/usr/bin/pseudo " % native_sysroot > > + > > + tarc_cmd = "tar cvpf %s/rootfs.tar -C %s ." % (cr_workdir, rootfs_dir) > > + rc, out = exec_native_cmd(pseudoc + tarc_cmd, native_sysroot) > > + > > + tarx_cmd = "tar xpvf %s/rootfs.tar -C %s" % (cr_workdir, > new_rootfs_dir) > > + rc, out = exec_native_cmd(pseudox + tarx_cmd, native_sysroot) > > + > > + rm_cmd = "rm %s/rootfs.tar" % cr_workdir > > + rc, out = exec_cmd(rm_cmd, True) > > + > > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > > + > > + return new_rootfs_dir > > + > > +class UBootPlugin(SourcePlugin): > > + name = 'uboot' > > + > > + @classmethod > > + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, > krootfs_dir, > > + bootimg_dir, kernel_dir, native_sysroot): > > + """ > > + Called before all partitions have been prepared and assembled into a > > + disk image. Intall packages based on wic configuration. > > + """ > > + > > Same here, this looks like general-purpose package installation code > that shouldn't be part of a u-boot plugin. Also, is that comment > correct? [Adrian Calianu] This method is part of a more generic pluginbase class and each source plugin has the option to generate rootfs from packages or not. It could happen to have more partitions and each partition with a specific rootfs. That specific rootfs can be generated (installing packages) or user specified (--rootfs) and this is controlled by source plugin. > > > + # set new rootfs_dir > > + rootfs_dir = create_local_rootfs(part, creator, cr_workdir, krootfs_dir, > native_sysroot) > > + > > + # wks file parsing > > + packages = kickstart.get_packages(creator.ks) > > + > > + # wic.conf file parsing = found under 'creator' > > + local_pkgs_path = creator._local_pkgs_path > > + repourl = creator.repourl > > + pkgmgr = creator.pkgmgr_name > > + > > + # install packages > > + if packages and pkgmgr in ["opkg"]: > > + if len(repourl) > 0 : > > + part.install_pkgs_ipk(cr_workdir, oe_builddir, rootfs_dir, > native_sysroot, > > + packages, repourl) > > + else: > > + msger.error("No packages repository provided in wic.conf") > > + > > + @classmethod > > + def do_prepare_partition(self, part, cr, cr_workdir, oe_builddir, > bootimg_dir, > > + kernel_dir, krootfs_dir, native_sysroot): > > + """ > > + Called to do the actual content population for a partition i.e. it > > + 'prepares' the partition to be incorporated into the image. > > + In this case, prepare content for legacy bios boot partition. > > + """ > > + if part.rootfs is None: > > + if not 'ROOTFS_DIR' in krootfs_dir: > > + msg = "Couldn't find --rootfs-dir, exiting" > > + msger.error(msg) > > + rootfs_dir = krootfs_dir['ROOTFS_DIR'] > > + else: > > + if part.rootfs in krootfs_dir: > > + rootfs_dir = krootfs_dir[part.rootfs] > > + elif os.path.isdir(part.rootfs): > > + rootfs_dir = part.rootfs > > + else: > > + msg = "Couldn't find --rootfs-dir=%s connection" > > + msg += " or it is not a valid path, exiting" > > + msger.error(msg % part.rootfs) > > + > > + part.set_rootfs(rootfs_dir) > > + > > + # change partition label wich will reflect into the final rootfs image > name > > + part.label = "%s_%s" % (part.label, cr.name) > > + > > + defpath = os.environ['PATH'] > > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > > + > > + part.prepare_rootfs(cr_workdir, oe_builddir, rootfs_dir, > native_sysroot) > > + part.prepare_for_uboot(cr.target_arch,cr_workdir, oe_builddir, > rootfs_dir, native_sysroot) > > + > > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > > diff --git a/scripts/lib/mic/utils/oe/package_manager.py > b/scripts/lib/mic/utils/oe/package_manager.py > > new file mode 100644 > > index 0000000..92ce98e > > --- /dev/null > > +++ b/scripts/lib/mic/utils/oe/package_manager.py > > @@ -0,0 +1,810 @@ > > +# ex:ts=4:sw=4:sts=4:et > > +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > > +# > > +# Copyright (c) 2014, Enea AB. > > +# All rights reserved. > > +# > > +# This program is free software; you can redistribute it and/or modify > > +# it under the terms of the GNU General Public License version 2 as > > +# published by the Free Software Foundation. > > +# > > +# This program is distributed in the hope that it will be useful, > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > +# GNU General Public License for more details. > > +# > > +# You should have received a copy of the GNU General Public License > along > > +# with this program; if not, write to the Free Software Foundation, Inc., > > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > > +# > > +# DESCRIPTION > > +# This implements the opkg package manager wrapper as a combination > of > > +# meta/lib/oe/package_manager.py and bitbake/lib/bb/utils.py files and > > +# adaptation of those files to 'wic'. > > +# > > +# AUTHORS > > +# Adrian Calianu <adrian.calianu (at] enea.com> > > +# > > +# This file incorporates work covered by the following copyright and > > +# permission notice: > > +# > > +# meta/COPYING.GPLv2 (GPLv2) > > +# meta/COPYING.MIT (MIT) > > +# > > +# Copyright (C) 2004 Michael Lauer > > +# > > +# Permission to use, copy, modify, and/or distribute this software > > +# for any purpose with or without fee is hereby granted, provided > > +# that the above copyright notice and this permission notice appear > > +# in all copies. > > +# > > +# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL > > +# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL > IMPLIED > > +# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT > SHALL THE > > +# AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR > > +# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER > RESULTING FROM LOSS > > +# OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, > > +# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN > > +# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. > > + > > + > > +from abc import ABCMeta, abstractmethod > > +import os > > +import glob > > +import subprocess > > +import shutil > > +import multiprocessing > > +import re > > +import errno > > +import fcntl > > + > > +from mic.utils.oe.misc import * > > +from mic import msger > > + > > +def mkdirhier(directory): > > + """Create a directory like 'mkdir -p', but does not complain if > > + directory already exists like os.makedirs > > + """ > > + > > + try: > > + os.makedirs(directory) > > + except OSError as e: > > + if e.errno != errno.EEXIST: > > + raise e > > + > > +def remove(path, recurse=False): > > + """Equivalent to rm -f or rm -rf""" > > + if not path: > > + return > > + if recurse: > > + # shutil.rmtree(name) would be ideal but its too slow > > + subprocess.call(['rm', '-rf'] + glob.glob(path)) > > + return > > + for name in glob.glob(path): > > + try: > > + os.unlink(name) > > + except OSError as exc: > > + if exc.errno != errno.ENOENT: > > + raise > > + > > +def lockfile(name, shared=False, retry=True): > > + """ > > + Use the file fn as a lock file, return when the lock has been acquired. > > + Returns a variable to pass to unlockfile(). > > + """ > > + dirname = os.path.dirname(name) > > + mkdirhier(dirname) > > + > > + if not os.access(dirname, os.W_OK): > > + logger.error("Unable to acquire lock '%s', directory is not writable", > > + name) > > + sys.exit(1) > > + > > + op = fcntl.LOCK_EX > > + if shared: > > + op = fcntl.LOCK_SH > > + if not retry: > > + op = op | fcntl.LOCK_NB > > + > > + while True: > > + # If we leave the lockfiles lying around there is no problem > > + # but we should clean up after ourselves. This gives potential > > + # for races though. To work around this, when we acquire the lock > > + # we check the file we locked was still the lock file on disk. > > + # by comparing inode numbers. If they don't match or the lockfile > > + # no longer exists, we start again. > > + > > + # This implementation is unfair since the last person to request the > > + # lock is the most likely to win it. > > + > > + try: > > + lf = open(name, 'a+') > > + fileno = lf.fileno() > > + fcntl.flock(fileno, op) > > + statinfo = os.fstat(fileno) > > + if os.path.exists(lf.name): > > + statinfo2 = os.stat(lf.name) > > + if statinfo.st_ino == statinfo2.st_ino: > > + return lf > > + lf.close() > > + except Exception: > > + try: > > + lf.close() > > + except Exception: > > + pass > > + pass > > + if not retry: > > + return None > > + > > +def unlockfile(lf): > > + """ > > + Unlock a file locked using lockfile() > > + """ > > + try: > > + # If we had a shared lock, we need to promote to exclusive before > > + # removing the lockfile. Attempt this, ignore failures. > > + fcntl.flock(lf.fileno(), fcntl.LOCK_EX|fcntl.LOCK_NB) > > + os.unlink(lf.name) > > + except (IOError, OSError): > > + pass > > + fcntl.flock(lf.fileno(), fcntl.LOCK_UN) > > + lf.close() > > + > > +def which(path, item, direction = 0, history = False): > > + """ > > + Locate a file in a PATH > > + """ > > + > > + hist = [] > > + paths = (path or "").split(':') > > + if direction != 0: > > + paths.reverse() > > + > > + for p in paths: > > + next = os.path.join(p, item) > > + hist.append(next) > > + if os.path.exists(next): > > + if not os.path.isabs(next): > > + next = os.path.abspath(next) > > + if history: > > + return next, hist > > + return next > > + > > + if history: > > + return "", hist > > + return "" > > + > > + > > + > > +# this can be used by all PM backends to create the index files in parallel > > +def wic_create_index(arg): > > + index_cmd = arg > > + > > + try: > > + msger.info("Executing '%s' ..." % index_cmd) > > + subprocess.check_output(index_cmd, stderr=subprocess.STDOUT, > shell=True) > > + except subprocess.CalledProcessError as e: > > + return("Index creation command '%s' failed with return code > %d:\n%s" % > > + (e.cmd, e.returncode, e.output)) > > + > > + return None > > + > > + > > +class WicIndexer(object): > > + __metaclass__ = ABCMeta > > + > > + def __init__(self, d, deploy_dir): > > + self.d = d > > + self.deploy_dir = deploy_dir > > + > > + @abstractmethod > > + def write_index(self): > > + pass > > + > > +class WicOpkgIndexer(WicIndexer): > > + def write_index(self): > > + arch_vars = ["ALL_MULTILIB_PACKAGE_ARCHS", > > + "SDK_PACKAGE_ARCHS", > > + "MULTILIB_ARCHS"] > > + > > + opkg_index_cmd = which(os.getenv('PATH'), "opkg-make-index") > > + > > + if not os.path.exists(os.path.join(self.deploy_dir, "Packages")): > > + open(os.path.join(self.deploy_dir, "Packages"), "w").close() > > + > > + index_cmds = [] > > + for arch_var in arch_vars: > > + if self.d.has_key(arch_var): > > + archs = self.d[arch_var] > > + else: > > + archs = None > > + > > + if archs is None: > > + continue > > + > > + for arch in archs.split(): > > + pkgs_dir = os.path.join(self.deploy_dir, arch) > > + pkgs_file = os.path.join(pkgs_dir, "Packages") > > + > > + if not os.path.isdir(pkgs_dir): > > + continue > > + > > + if not os.path.exists(pkgs_file): > > + open(pkgs_file, "w").close() > > + > > + index_cmds.append('%s -r %s -p %s -m %s' % > > + (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir)) > > + > > + if len(index_cmds) == 0: > > + msger.info("There are no packages in %s!" % self.deploy_dir) > > + return > > + > > + nproc = multiprocessing.cpu_count() > > + pool = multiprocessing.Pool(nproc) > > + results = list(pool.imap(wic_create_index, index_cmds)) > > + pool.close() > > + pool.join() > > + > > + for result in results: > > + if result is not None: > > + return(result) > > + > > +class WicPkgsList(object): > > + __metaclass__ = ABCMeta > > + > > + def __init__(self, d, rootfs_dir): > > + self.d = d > > + self.rootfs_dir = rootfs_dir > > + > > + @abstractmethod > > + def list(self, format=None): > > + pass > > + > > + > > +class WicOpkgPkgsList(WicPkgsList): > > + def __init__(self, d, rootfs_dir, config_file): > > + super(WicOpkgPkgsList, self).__init__(d, rootfs_dir) > > + > > + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") > > + self.opkg_args = "-f %s -o %s " % (config_file, rootfs_dir) > > + if self.d.has_key("OPKG_ARGS"): > > + self.opkg_args += self.d["OPKG_ARGS"] > > + > > + def list(self, format=None): > > + opkg_query_cmd = which(os.getenv('PATH'), "opkg-query- > helper.py") > > + > > + if format == "arch": > > + cmd = "%s %s status | %s -a" % \ > > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > > + elif format == "file": > > + cmd = "%s %s status | %s -f" % \ > > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > > + elif format == "ver": > > + cmd = "%s %s status | %s -v" % \ > > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > > + elif format == "deps": > > + cmd = "%s %s status | %s" % \ > > + (self.opkg_cmd, self.opkg_args, opkg_query_cmd) > > + else: > > + cmd = "%s %s list_installed | cut -d' ' -f1" % \ > > + (self.opkg_cmd, self.opkg_args) > > + > > + try: > > + output = subprocess.check_output(cmd, > stderr=subprocess.STDOUT, shell=True).strip() > > + except subprocess.CalledProcessError as e: > > + msger.error("Cannot get the installed packages list. Command '%s' " > > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > > + > > + if output and format == "file": > > + tmp_output = "" > > + for line in output.split('\n'): > > + pkg, pkg_file, pkg_arch = line.split() > > + full_path = os.path.join(self.rootfs_dir, pkg_arch, pkg_file) > > + if os.path.exists(full_path): > > + tmp_output += "%s %s %s\n" % (pkg, full_path, pkg_arch) > > + else: > > + tmp_output += "%s %s %s\n" % (pkg, pkg_file, pkg_arch) > > + > > + output = tmp_output > > + > > + return output > > + > > + > > +class WicPackageManager(object): > > + """ > > + This is an abstract class. Do not instantiate this directly. > > + """ > > + __metaclass__ = ABCMeta > > + > > + def __init__(self, d, pseudo, native_sysroot): > > + self.d = d > > + self.deploy_dir = None > > + self.deploy_lock = None > > + if self.d.has_key('PACKAGE_FEED_URIS'): > > + self.feed_uris = self.d['PACKAGE_FEED_URIS'] > > + else: > > + self.feed_uris = "" > > + self.pseudo = pseudo > > + self.native_sysroot = native_sysroot > > + > > + """ > > + Update the package manager package database. > > + """ > > + @abstractmethod > > + def update(self): > > + pass > > + > > + """ > > + Install a list of packages. 'pkgs' is a list object. If 'attempt_only' is > > + True, installation failures are ignored. > > + """ > > + @abstractmethod > > + def install(self, pkgs, attempt_only=False): > > + pass > > + > > + """ > > + Remove a list of packages. 'pkgs' is a list object. If 'with_dependencies' > > + is False, the any dependencies are left in place. > > + """ > > + @abstractmethod > > + def remove(self, pkgs, with_dependencies=True): > > + pass > > + > > + """ > > + This function creates the index files > > + """ > > + @abstractmethod > > + def write_index(self): > > + pass > > + > > + @abstractmethod > > + def remove_packaging_data(self): > > + pass > > + > > + @abstractmethod > > + def list_installed(self, format=None): > > + pass > > + > > + @abstractmethod > > + def insert_feeds_uris(self): > > + pass > > + > > + """ > > + Install complementary packages based upon the list of currently > installed > > + packages e.g. locales, *-dev, *-dbg, etc. This will only attempt to install > > + these packages, if they don't exist then no error will occur. Note: every > > + backend needs to call this function explicitly after the normal package > > + installation > > + """ > > + def install_complementary(self, globs=None): > > + # we need to write the list of installed packages to a file because the > > + # oe-pkgdata-util reads it from a file > > + if self.d.has_key('WORKDIR'): > > + installed_pkgs_file = os.path.join(self.d['WORKDIR'], > > + "installed_pkgs.txt") > > + else: > > + msger.error("No WORKDIR provided!") > > + > > + with open(installed_pkgs_file, "w+") as installed_pkgs: > > + installed_pkgs.write(self.list_installed("arch")) > > + > > + if globs is None: > > + if self.d.has_key('IMAGE_INSTALL_COMPLEMENTARY'): > > + globs = self.d['IMAGE_INSTALL_COMPLEMENTARY'] > > + split_linguas = set() > > + > > + if self.d.has_key('IMAGE_LINGUAS'): > > + for translation in self.d['IMAGE_LINGUAS'].split(): > > + split_linguas.add(translation) > > + split_linguas.add(translation.split('-')[0]) > > + > > + split_linguas = sorted(split_linguas) > > + > > + for lang in split_linguas: > > + globs += " *-locale-%s" % lang > > + > > + if globs is None: > > + return > > + > > + if not self.d.has_key('PKGDATA_DIR'): > > + msger.error("No PKGDATA_DIR provided!") > > + > > + cmd = [which(os.getenv('PATH'), "oe-pkgdata-util"), > > + "glob", self.d['PKGDATA_DIR'], installed_pkgs_file, > > + globs] > > + > > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > > + if rc != 0: > > + msger.error("Could not compute complementary packages list. > Command " > > + "'%s' returned %d" % > > + (' '.join(cmd), rc)) > > + > > + self.install(out.split(), attempt_only=True) > > + > > + > > + def deploy_dir_lock(self): > > + if self.deploy_dir is None: > > + raise RuntimeError("deploy_dir is not set!") > > + > > + lock_file_name = os.path.join(self.deploy_dir, "deploy.lock") > > + > > + self.deploy_lock = lockfile(lock_file_name) > > + > > + def deploy_dir_unlock(self): > > + if self.deploy_lock is None: > > + return > > + > > + unlockfile(self.deploy_lock) > > + > > + self.deploy_lock = None > > + > > + > > +class WicOpkgPM(WicPackageManager): > > + def __init__(self, d, target_rootfs, config_file, archs, pseudo, > native_sysroot, task_name='target'): > > + super(WicOpkgPM, self).__init__(d, pseudo, native_sysroot) > > + > > + self.target_rootfs = target_rootfs > > + self.config_file = config_file > > + self.pkg_archs = archs > > + self.task_name = task_name > > + > > + if self.d.has_key("DEPLOY_DIR_IPK"): > > + self.deploy_dir = self.d["DEPLOY_DIR_IPK"] > > + > > + self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock") > > + self.opkg_cmd = which(os.getenv('PATH'), "opkg-cl") > > + self.opkg_args = "-f %s -o %s " % (self.config_file, target_rootfs) > > + if self.d.has_key("OPKG_ARGS"): > > + self.opkg_args += self.d["OPKG_ARGS"] > > + > > + if self.d.has_key('OPKGLIBDIR'): > > + opkg_lib_dir = self.d['OPKGLIBDIR'] > > + else: > > + opkg_lib_dir = "" > > + > > + if opkg_lib_dir[0] == "/": > > + opkg_lib_dir = opkg_lib_dir[1:] > > + > > + self.opkg_dir = os.path.join(target_rootfs, opkg_lib_dir, "opkg") > > + > > + mkdirhier(self.opkg_dir) > > + > > + if self.d.has_key("TMPDIR"): > > + tmp_dir = self.d["TMPDIR"] > > + else: > > + tmp_dir = "" > > + > > + self.saved_opkg_dir = '%s/saved/%s' % (tmp_dir, self.task_name) > > + if not os.path.exists('%s/saved' % tmp_dir): > > + mkdirhier('%s/saved' % tmp_dir) > > + > > + if self.d.has_key('BUILD_IMAGES_FROM_FEEDS') and > self.d['BUILD_IMAGES_FROM_FEEDS'] != "1": > > + self._create_config() > > + else: > > + self._create_custom_config() > > + > > + self.indexer = WicOpkgIndexer(self.d, self.deploy_dir) > > + > > + """ > > + This function will change a package's status in /var/lib/opkg/status file. > > + If 'packages' is None then the new_status will be applied to all > > + packages > > + """ > > + def mark_packages(self, status_tag, packages=None): > > + status_file = os.path.join(self.opkg_dir, "status") > > + > > + with open(status_file, "r") as sf: > > + with open(status_file + ".tmp", "w+") as tmp_sf: > > + if packages is None: > > + tmp_sf.write(re.sub(r"Package: (.*?)\n((?:[^\n]+\n)*?)Status: > (.*)(?:unpacked|installed)", > > + r"Package: \1\n\2Status: \3%s" % status_tag, > > + sf.read())) > > + else: > > + if type(packages).__name__ != "list": > > + raise TypeError("'packages' should be a list object") > > + > > + status = sf.read() > > + for pkg in packages: > > + status = re.sub(r"Package: %s\n((?:[^\n]+\n)*?)Status: > (.*)(?:unpacked|installed)" % pkg, > > + r"Package: %s\n\1Status: \2%s" % (pkg, status_tag), > > + status) > > + > > + tmp_sf.write(status) > > + > > + os.rename(status_file + ".tmp", status_file) > > + > > + def _create_custom_config(self): > > + msger.info("Building from feeds activated!") > > + > > + with open(self.config_file, "w+") as config_file: > > + priority = 1 > > + for arch in self.pkg_archs.split(): > > + config_file.write("arch %s %d\n" % (arch, priority)) > > + priority += 5 > > + > > + if self.d.has_key('IPK_FEED_URIS'): > > + ipk_feed_uris = self.d['IPK_FEED_URIS'] > > + else: > > + ipk_feed_uris = "" > > + > > + for line in ipk_feed_uris.split(): > > + feed_match = re.match("^[ \t]*(.*)##([^ \t]*)[ \t]*$", line) > > + > > + if feed_match is not None: > > + feed_name = feed_match.group(1) > > + feed_uri = feed_match.group(2) > > + > > + msger.info("Add %s feed with URL %s" % (feed_name, > feed_uri)) > > + > > + config_file.write("src/gz %s %s\n" % (feed_name, feed_uri)) > > + > > + """ > > + Allow to use package deploy directory contents as quick devel- > testing > > + feed. This creates individual feed configs for each arch subdir of > those > > + specified as compatible for the current machine. > > + NOTE: Development-helper feature, NOT a full-fledged feed. > > + """ > > + if self.d.has_key('FEED_DEPLOYDIR_BASE_URI'): > > + feed_deploydir_base_dir = self.d['FEED_DEPLOYDIR_BASE_URI'] > > + else: > > + feed_deploydir_base_dir = "" > > + > > + if feed_deploydir_base_dir != "": > > + for arch in self.pkg_archs.split(): > > + if self.d.has_key("sysconfdir"): > > + sysconfdir = self.d["sysconfdir"] > > + else: > > + sysconfdir = None > > + > > + cfg_file_name = os.path.join(self.target_rootfs, > > + sysconfdir, > > + "opkg", > > + "local-%s-feed.conf" % arch) > > + > > + with open(cfg_file_name, "w+") as cfg_file: > > + cfg_file.write("src/gz local-%s %s/%s" % > > + arch, > > + feed_deploydir_base_dir, > > + arch) > > + > > + def _create_config(self): > > + with open(self.config_file, "w+") as config_file: > > + priority = 1 > > + for arch in self.pkg_archs.split(): > > + config_file.write("arch %s %d\n" % (arch, priority)) > > + priority += 5 > > + > > + config_file.write("src oe file:%s\n" % self.deploy_dir) > > + > > + for arch in self.pkg_archs.split(): > > + pkgs_dir = os.path.join(self.deploy_dir, arch) > > + if os.path.isdir(pkgs_dir): > > + config_file.write("src oe-%s file:%s\n" % > > + (arch, pkgs_dir)) > > + > > + def insert_feeds_uris(self): > > + if self.feed_uris == "": > > + return > > + > > + rootfs_config = os.path.join('%s/etc/opkg/base-feeds.conf' > > + % self.target_rootfs) > > + > > + with open(rootfs_config, "w+") as config_file: > > + uri_iterator = 0 > > + for uri in self.feed_uris.split(): > > + config_file.write("src/gz url-%d %s/ipk\n" % > > + (uri_iterator, uri)) > > + > > + for arch in self.pkg_archs.split(): > > + if not os.path.exists(os.path.join(self.deploy_dir, arch)): > > + continue > > + msger.info('Note: adding opkg channel url-%s-%d (%s)' % > > + (arch, uri_iterator, uri)) > > + > > + config_file.write("src/gz uri-%s-%d %s/ipk/%s\n" % > > + (arch, uri_iterator, uri, arch)) > > + uri_iterator += 1 > > + > > + def update(self): > > + self.deploy_dir_lock() > > + > > + cmd = "%s %s update" % (self.opkg_cmd, self.opkg_args) > > + > > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > > + if rc != 0: > > + self.deploy_dir_unlock() > > + msger.error("Unable to update the package index files. Command > '%s' " > > + "returned %d" % (cmd, rc)) > > + > > + self.deploy_dir_unlock() > > + > > + def install(self, pkgs, attempt_only=False): > > + if attempt_only and len(pkgs) == 0: > > + return > > + > > + cmd = "%s %s install %s" % (self.opkg_cmd, self.opkg_args, ' > '.join(pkgs)) > > + > > + os.environ['D'] = self.target_rootfs > > + os.environ['OFFLINE_ROOT'] = self.target_rootfs > > + os.environ['IPKG_OFFLINE_ROOT'] = self.target_rootfs > > + os.environ['OPKG_OFFLINE_ROOT'] = self.target_rootfs > > + if self.d.has_key('WORKDIR'): > > + os.environ['INTERCEPT_DIR'] = os.path.join(self.d['WORKDIR'], > > + "intercept_scripts") > > + else: > > + os.environ['INTERCEPT_DIR'] = "." > > + msger.warning("No WORKDIR provided!") > > + > > + if self.d.has_key('STAGING_DIR_NATIVE'): > > + os.environ['NATIVE_ROOT'] = self.d['STAGING_DIR_NATIVE'] > > + else: > > + msger.error("No STAGING_DIR_NATIVE provided!") > > + > > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > > + if rc != 0: > > + msger.error("Unable to install packages. " > > + "Command '%s' returned %d" % (cmd, rc)) > > + > > + > > + def remove(self, pkgs, with_dependencies=True): > > + if with_dependencies: > > + cmd = "%s %s --force-depends --force-remove --force-removal-of- > dependent-packages remove %s" % \ > > + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) > > + else: > > + cmd = "%s %s --force-depends remove %s" % \ > > + (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) > > + > > + rc, out = exec_native_cmd(self.pseudo + cmd, self.native_sysroot) > > + if rc != 0: > > + msger.error("Unable to remove packages. Command '%s' " > > + "returned %d" % (cmd, rc)) > > + > > + > > + def write_index(self): > > + self.deploy_dir_lock() > > + > > + result = self.indexer.write_index() > > + > > + self.deploy_dir_unlock() > > + > > + if result is not None: > > + msger.error(result) > > + > > + def remove_packaging_data(self): > > + remove(self.opkg_dir, True) > > + # create the directory back, it's needed by PM lock > > + mkdirhier(self.opkg_dir) > > + > > + def list_installed(self, format=None): > > + return WicOpkgPkgsList(self.d, self.target_rootfs, > self.config_file).list(format) > > + > > + def handle_bad_recommendations(self): > > + if self.d.has_key("BAD_RECOMMENDATIONS"): > > + bad_recommendations = self.d["BAD_RECOMMENDATIONS"] > > + else: > > + bad_recommendations = "" > > + > > + if bad_recommendations.strip() == "": > > + return > > + > > + status_file = os.path.join(self.opkg_dir, "status") > > + > > + # If status file existed, it means the bad recommendations has already > > + # been handled > > + if os.path.exists(status_file): > > + return > > + > > + cmd = "%s %s info " % (self.opkg_cmd, self.opkg_args) > > + > > + with open(status_file, "w+") as status: > > + for pkg in bad_recommendations.split(): > > + pkg_info = cmd + pkg > > + > > + try: > > + output = subprocess.check_output(pkg_info.split(), > stderr=subprocess.STDOUT).strip() > > + except subprocess.CalledProcessError as e: > > + msger.error("Cannot get package info. Command '%s' " > > + "returned %d:\n%s" % (pkg_info, e.returncode, e.output)) > > + > > + if output == "": > > + msger.info("Ignored bad recommendation: '%s' is " > > + "not a package" % pkg) > > + continue > > + > > + for line in output.split('\n'): > > + if line.startswith("Status:"): > > + status.write("Status: deinstall hold not-installed\n") > > + else: > > + status.write(line + "\n") > > + > > + ''' > > + The following function dummy installs pkgs and returns the log of > output. > > + ''' > > + def dummy_install(self, pkgs): > > + if len(pkgs) == 0: > > + return > > + > > + # Create an temp dir as opkg root for dummy installation > > + if self.d.has_key("TMPDIR"): > > + tmp_dir = self.d["TMPDIR"] > > + else: > > + tmp_dir = "." > > + msger.warning("No TMPDIR provided!") > > + > > + temp_rootfs = '%s/opkg' % tmp_dir > > + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') > > + mkdirhier(temp_opkg_dir) > > + > > + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) > > + if self.d.has_key("OPKG_ARGS"): > > + opkg_args += self.d["OPKG_ARGS"] > > + > > + cmd = "%s %s update" % (self.opkg_cmd, opkg_args) > > + try: > > + subprocess.check_output(cmd, stderr=subprocess.STDOUT, > shell=True) > > + except subprocess.CalledProcessError as e: > > + msger.error("Unable to update. Command '%s' " > > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > > + > > + # Dummy installation > > + cmd = "%s %s --noaction install %s " % (self.opkg_cmd, > > + opkg_args, > > + ' '.join(pkgs)) > > + try: > > + output = subprocess.check_output(cmd, > stderr=subprocess.STDOUT, shell=True) > > + except subprocess.CalledProcessError as e: > > + msger.error("Unable to dummy install packages. Command '%s' " > > + "returned %d:\n%s" % (cmd, e.returncode, e.output)) > > + > > + remove(temp_rootfs, True) > > + > > + return output > > + > > + def backup_packaging_data(self): > > + # Save the opkglib for increment ipk image generation > > + if os.path.exists(self.saved_opkg_dir): > > + remove(self.saved_opkg_dir, True) > > + shutil.copytree(self.opkg_dir, > > + self.saved_opkg_dir, > > + symlinks=True) > > + > > + def recover_packaging_data(self): > > + # Move the opkglib back > > + if os.path.exists(self.saved_opkg_dir): > > + if os.path.exists(self.opkg_dir): > > + remove(self.opkg_dir, True) > > + > > + msger.info('Recover packaging data') > > + shutil.copytree(self.saved_opkg_dir, > > + self.opkg_dir, > > + symlinks=True) > > + > > + > > +def wic_generate_index_files(d): > > + if d.has_key('PACKAGE_CLASSES'): > > + classes = d['PACKAGE_CLASSES'].replace("package_", "").split() > > + else: > > + classes = "" > > + msger.warning("No PACKAGE_CLASSES provided!") > > + > > + if d.has_key('DEPLOY_DIR_IPK'): > > + deploy_dir_ipk = d['DEPLOY_DIR_IPK'] > > + else: > > + deploy_dir_ipk = None > > + msger.warning("No DEPLOY_DIR_IPK provided!") > > + > > + indexer_map = { > > + "ipk": (WicOpkgIndexer, deploy_dir_ipk) > > + } > > + > > + result = None > > + > > + for pkg_class in classes: > > + if not pkg_class in indexer_map: > > + continue > > + > > + if os.path.exists(indexer_map[pkg_class][1]): > > + result = indexer_map[pkg_class][0](d, > indexer_map[pkg_class][1]).write_index() > > + > > + if result is not None: > > + msger.error(result) > > diff --git a/scripts/wic b/scripts/wic > > index 2d3fd09..b9c8756 100755 > > --- a/scripts/wic > > +++ b/scripts/wic > > @@ -99,6 +99,10 @@ def wic_create_subcommand(args, usage_str): > > > > (options, args) = parser.parse_args(args) > > > > + if options.debug: > > + loglevel = logging.DEBUG > > + start_logging(loglevel) > > + > > if len(args) != 1: > > logging.error("Wrong number of arguments, exiting\n") > > parser.print_help() > > @@ -107,9 +111,11 @@ def wic_create_subcommand(args, usage_str): > > if not options.image_name and not (options.rootfs_dir and > > options.bootimg_dir and > > options.kernel_dir and > > + options.native_sysroot or > > options.native_sysroot): > > print "Build artifacts not completely specified, exiting." > > - print " (Use 'wic -e' or 'wic -r -b -k -n' to specify artifacts)" > > + print " (Use 'wic -e' or 'wic -r -b -k -n' or 'wic -r -n' to specify artifacts)" > > + print options > > sys.exit(1) > > > > if not options.image_name: > > @@ -125,13 +131,16 @@ def wic_create_subcommand(args, usage_str): > > > > print "Creating image(s)...\n" > > > > - bitbake_env_lines = find_bitbake_env_lines(options.image_name) > > - if not bitbake_env_lines: > > - print "Couldn't get bitbake environment, exiting." > > - sys.exit(1) > > - set_bitbake_env_lines(bitbake_env_lines) > > + # If '-e' option is used the values are extracted from bitbake env. > > + if options.image_name: > > + bitbake_env_lines = find_bitbake_env_lines(options.image_name) > > + if not bitbake_env_lines: > > + print "Couldn't get bitbake environment, exiting." > > + sys.exit(1) > > + set_bitbake_env_lines(bitbake_env_lines) > > > > bootimg_dir = staging_data_dir = hdddir = "" > > + rootfs_dir = native_sysroot = kernel_dir = image_output_dir = "" > > > > if options.image_name: > > (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot) = \ > > @@ -140,34 +149,65 @@ def wic_create_subcommand(args, usage_str): > > wks_file = args[0] > > > > if not wks_file.endswith(".wks"): > > + # Return full path of the .wks file > > wks_file = find_canned_image(scripts_path, wks_file) > > if not wks_file: > > - print "No image named %s found, exiting. (Use 'wic list images' to > list available images, or specify a fully-qualified OE kickstart (.wks) > filename)\n" % wks_file > > + print "No image named %s found, exiting.\n" % wks_file > > + print "(Use 'wic list images' to list available images, or specify a fully- > qualified OE kickstart (.wks) filename)\n" > > sys.exit(1) > > > > - image_output_dir = "" > > if options.outdir: > > image_output_dir = options.outdir > > > > - if not options.image_name: > > - rootfs_dir = '' > > - if 'ROOTFS_DIR' in options.rootfs_dir: > > - rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > > - bootimg_dir = options.bootimg_dir > > - kernel_dir = options.kernel_dir > > + if options.native_sysroot: > > native_sysroot = options.native_sysroot > > - if rootfs_dir and not os.path.isdir(rootfs_dir): > > - print "--roofs-dir (-r) not found, exiting\n" > > - sys.exit(1) > > - if not os.path.isdir(bootimg_dir): > > - print "--bootimg-dir (-b) not found, exiting\n" > > - sys.exit(1) > > - if not os.path.isdir(kernel_dir): > > - print "--kernel-dir (-k) not found, exiting\n" > > - sys.exit(1) > > + print "Using native_sysroot from user command: %s" % > native_sysroot > > + > > if not os.path.isdir(native_sysroot): > > - print "--native-sysroot (-n) not found, exiting\n" > > + print "--native-sysroot (-n) not found, exiting" > > sys.exit(1) > > + > > + native_sysroot = os.path.abspath(native_sysroot) > > + > > + if not options.image_name: > > + if (options.bootimg_dir and options.kernel_dir and > > + options.rootfs_dir and options.native_sysroot): > > + rootfs_dir = '' > > + if 'ROOTFS_DIR' in options.rootfs_dir: > > + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > > + bootimg_dir = options.bootimg_dir > > + kernel_dir = options.kernel_dir > > + native_sysroot = options.native_sysroot > > + > > + if rootfs_dir and not os.path.isdir(rootfs_dir): > > + print "--roofs-dir (-r) not found, exiting\n" > > + sys.exit(1) > > + if not os.path.isdir(bootimg_dir): > > + print "--bootimg-dir (-b) not found, exiting\n" > > + sys.exit(1) > > + if not os.path.isdir(kernel_dir): > > + print "--kernel-dir (-k) not found, exiting\n" > > + sys.exit(1) > > + if not os.path.isdir(native_sysroot): > > + print "--native-sysroot (-n) not found, exiting\n" > > + sys.exit(1) > > + else: > > + print 'Build image from rootfs and a package list using native > rootfs\n' > > + if options.rootfs_dir and 'ROOTFS_DIR' in options.rootfs_dir: > > + rootfs_dir = options.rootfs_dir['ROOTFS_DIR'] > > + elif options.rootfs_dir: > > + rootfs_dir = options.rootfs_dir > > + else: > > + rootfs_dir = "" > > + > > + native_sysroot = options.native_sysroot > > + > > + if rootfs_dir and not os.path.isdir(rootfs_dir): > > + print "--roofs-dir (-r) not found, exiting\n" > > + sys.exit(1) > > + if not os.path.isdir(native_sysroot): > > + print "--native-sysroot (-n) not found, exiting\n" > > + sys.exit(1) > > else: > > not_found = not_found_dir = "" > > if not os.path.isdir(rootfs_dir): > > -- > > 1.7.10.4 > > > > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv2 1/2] Add wic support to generate rootfs image for uboot 2014-06-16 13:18 ` Adrian Calianu @ 2014-06-17 9:35 ` Anders Darander 2014-06-20 14:46 ` Tom Zanussi 1 sibling, 0 replies; 11+ messages in thread From: Anders Darander @ 2014-06-17 9:35 UTC (permalink / raw) To: openembedded-core * Adrian Calianu <Adrian.Calianu@enea.com> [140616 15:19]: > > As for the patch itself, it's huge and needs to be broken up into > > smaller pieces. In fact, it seems to be a combination of two > > completely separate things at least - a new mechanism for creating a > > rootfs from packages, and a new plugin for creating uboot partitions > > (which just happens to use the output from the new packaging step. > > So at minimum I think you need to break it up along the lines of > > that functionality, and it would be good to break it up even further > > e.g. 'Add a new do_pkg interface hook', 'Add an implementation of > > do_pkg for ipk', 'Add a new source plugin for uboot partitions', > > 'Add a new canned wks for a default uboot image', etc. The more you > > can break it up into logical chunks without going overboard, the > > easier it is to review and apply/revert. > [Adrian Calianu] I agree that there are two main features but the > reason of not break it in small pieces now was to provide a fully > functional patch for community to test it and be able to provide a > feedback for this patch, since those features cannot be tested > independently now. So, is it acceptable to provide patches that > cannot be tested? Well, if you split this into two or more patches, and you then submit this as a patch series, then I don't see how they couldn't be tested? Remember, no one forces you to send single patches. So I'd say, try to split them up. That'll make them easier to review, comment on, and to test. Cheers, Anders -- Anders Darander ChargeStorm AB / eStorm AB ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv2 1/2] Add wic support to generate rootfs image for uboot 2014-06-16 13:18 ` Adrian Calianu 2014-06-17 9:35 ` Anders Darander @ 2014-06-20 14:46 ` Tom Zanussi 1 sibling, 0 replies; 11+ messages in thread From: Tom Zanussi @ 2014-06-20 14:46 UTC (permalink / raw) To: Adrian Calianu Cc: Rifenbark, Scott M, Otavio Salvador, openembedded-core@lists.openembedded.org On Mon, 2014-06-16 at 13:18 +0000, Adrian Calianu wrote: > Hi Tom, > > First of all, thanks for your review comments! I'm glad that community has interest for this features. > I will answer to your comments in-line below... > > > -----Original Message----- > > From: openembedded-core-bounces@lists.openembedded.org > > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf > > Of Tom Zanussi > > Sent: Friday, June 13, 2014 11:19 PM > > To: y@enea.se > > Cc: openembedded-core@lists.openembedded.org; Otavio Salvador > > Subject: Re: [OE-core] [PATCHv2 1/2] Add wic support to generate rootfs > > image for uboot > > > > Hi, > > > > This looks like some nice new functionality, but I have some suggestions > > regarding the implementation and the posting itself... > > > > First, this seems to have been sent by a 'y' at enea, but they're > > apparently written by adrian.calianu at enea - why can't adrian.calianu > > send them himself? (makes me wonder about the authorship, etc). > [Adrian Calianu] It was an issue with local git server since the patch was send directly from git but nothing to worry about, problem was fixed. > > > > > Also, there seems to be just one big patch, but the subject says > > '[PATCHv2 1/2] Add wic support to generate rootfs image for uboot' - am > > I missing 2/2? > [Adrian Calianu] Actually there were two versions of patch and for > each patch a cover letter was also generated which lead to those > numbers. > > > > > As for the patch itself, it's huge and needs to be broken up into > > smaller pieces. In fact, it seems to be a combination of two completely > > separate things at least - a new mechanism for creating a rootfs from > > packages, and a new plugin for creating uboot partitions (which just > > happens to use the output from the new packaging step. So at minimum I > > think you need to break it up along the lines of that functionality, and > > it would be good to break it up even further e.g. 'Add a new do_pkg > > interface hook', 'Add an implementation of do_pkg for ipk', 'Add a new > > source plugin for uboot partitions', 'Add a new canned wks for a default > > uboot image', etc. The more you can break it up into logical chunks > > without going overboard, the easier it is to review and apply/revert. > [Adrian Calianu] I agree that there are two main features but the > reason of not break it in small pieces now was to provide a fully > functional patch for community to test it and be able to provide a > feedback for this patch, since those features cannot be tested > independently now. > So, is it acceptable to provide patches that cannot be tested? > Are you saying that the new packaging functionality is only useful for uboot users and that uboot support can't be done without the new packaging functionality? In any case, the normal way to submit a large patchset is to break it up into smaller reviewable pieces that hopefully build on each other. If the suggestions I mentioned above don't help, you should be able look in the archives for more concrete examples. > > > > Finally, something like this really needs to be documented in the tool - > > please add the relevant interface documentation to the help system. > [Adrian Calianu] Ok, I will investigate and add more info to help system. > I saw that the Yocto manual included wic doc only in 1.5.2 > version(http://www.yoctoproject.org/docs/1.5.2/dev-manual/dev-manual.html). Why that info is not found into the latest Yocto manual? > It is anything planned about that? > I'm not sure why the wic help was removed from post-1.5.2, cc'ing ScottR for that. > > > > For the 'packaging' part of this patch, the big problem I see is that > > the package_manager seems to be basically a copy of the oe package > > manager and more (which you mention in the comments) - I don't think it > > makes much sense to do that - wouldn't it be better to patch the oe > > package_manager code so that it could be reused by this tool? > [Adrian Calianu] This is a subject of discussion I think. Yes, first > time I just used the modules from bitbake and oe and worked perfectly > but this means to link wic tool by bitbake and all his specific > features like BB_variables, dataSmart,... Is this acceptable? > Shouldn't wic tool be as independent as possible? What is the plan for > wic tool in future? Yeah, wic is meant to be a standalone tool and should be completely independent of bitbake and oe. But it also doesn't make sense to copy bitbake and oe wholesale with minor changes into wic. Ideally you'd want a common library that both could make use of and layer the application-specific things on top of - would this be too large a project in this case? > > I see that the wic tool came with backend plugins (yum package > manager). So depending on the plans for wic tool in future we may > move this new package manager into backend plugin folder or use the > one from bitbake also for rpm packages and remove existing one. I see > it as an open subject. > Right, wic is built on top of mic which was built on top of Fedora livecd, etc, and instead of removing things like the yum package manager from the code, I left it in as possibly useful for unforeseen functionality like this. So if it makes sense to use that or plug into it or whatever, that's fine, but I'm guessing the oe-core stuff is more tailored to the needs of image creation for this project. Whatever makes the most sense. > > > > For the 'uboot' part, is the 'uboot' plugin implementing common > > functionality that all uboot images need, or is it for a specific use > > case (if so, it shouldn't be called 'uboot' but something more > > specific)? Otavio seemed to suggest that in general it would need to be > > modified for specific cases. I'm not sure about the best way to do that > > - we don't have many good examples yet. Can that sort of thing be > > parameterized/subclassed, or would it require a per-image plugin, which > > would be good to avoid. Not something you have to solve now, if the > > 'uboot' plugin is useful on its own and maybe subclassed/added onto by > > future plugins. It would be nice to have some specific and succinct > > examples to help think through what might be needed here. > [Adrian Calianu] I was thinking it as a general uboot plugin and in > future parameterized it like other existing plugins. We have plans to > extend it for SD/MMC cards. > OK, so that's what I was asking about, some idea on your plans to extend it. > > > > Some more specific comments in-line below... > > > > On Thu, 2014-06-12 at 10:43 +0200, y@enea.se wrote: > > > From: Adrian Calianu <adrian.calianu@enea.com> > > > > > > Add support in wic to build ramdisk uboot images from list of packages or > > > from an existing rootfs folder. > > > > > > Providing the list of packages starting with XXX-core-boot package(into wks > > file) > > > will generate an rootfs image for uboot out of bitbake. > > > > > > > There is also possible to combine an existing/application rootfs with a list > > > of packages that can be installed but the user must take care to provide > > > a basic core boot rootfs or XXX-core-boot package and reporul(wic.conf). > > > > > > > Is this implemented in the current patch? > [Adrian Calianu] Yes. This can be achieved by providing a valid basic > core boot package(into wic.conf) and user rootfs(wic create -r > "path_to_user_rootfs" -n "path_to_native_rootfs" ). > Please describe this in documentation (see below) - it's still not clear to me how this works. > > > > > Some prerequisites are required for this new build setup: > > > ../poky/scripts/lib/image/config/wic.conf > > > [create] > > > arch=target_arch (example: powerpc, arm, x86) > > > pkgmgr=opkg > > > repourl=http://example.distro/p2020rdb/ipk/all > > http://example.distro/p2020rdb/ipk/p2020rdb > > http://example.distro/p2020rdb/ipk/ppce500v2 > > > > > > 1) Build an rootfs image from an existing bitbake build: > > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core- > > image-minimal -o .../output > > > > > > 2) Build an rootfs image from an existing rootfs and native_sysroot: > > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r > > .../path_to_rootfs -n .../path_to_native_sysroot -o .../output > > > > > > 3) Build an rootfs image only from a package list (on wks file): > > > > > > Add the package list to be installed as rootfs on > > ../poky/scripts/lib/image/canned-wks/uboot.wks: > > > %packages > > > packagegroup-core-boot > > > pramfs-init > > > run-postinsts > > > packagegroup-core-ssh-dropbear > > > %end > > > > > > Generate rootfs image: > > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n > > .../path_to_native_sysroot -o .../output > > > > > > Signed-off-by: Adrian Calianu <adrian.calianu@enea.com> > > > --- > > > scripts/lib/image/canned-wks/uboot.wks | 17 + > > > scripts/lib/image/config/wic.conf | 4 + > > > scripts/lib/mic/imager/direct.py | 4 + > > > .../lib/mic/kickstart/custom_commands/partition.py | 128 ++++ > > > scripts/lib/mic/pluginbase.py | 9 + > > > scripts/lib/mic/plugins/source/uboot.py | 173 +++++ > > > scripts/lib/mic/utils/oe/package_manager.py | 810 > > ++++++++++++++++++++ > > > scripts/wic | 88 ++- > > > 10 files changed, 1219 insertions(+), 32 deletions(-) > > > create mode 100644 scripts/lib/image/canned-wks/uboot.wks > > > create mode 100644 scripts/lib/mic/plugins/source/uboot.py > > > create mode 100644 scripts/lib/mic/utils/oe/package_manager.py > > > > > > diff --git a/scripts/lib/image/canned-wks/uboot.wks > > b/scripts/lib/image/canned-wks/uboot.wks > > > new file mode 100644 > > > index 0000000..7de0572 > > > --- /dev/null > > > +++ b/scripts/lib/image/canned-wks/uboot.wks > > > @@ -0,0 +1,17 @@ > > > +# short-description: . Create a ramdisk image for U-Boot > > > +# long-description: Creates a ramdisk image for U-Boot that user > > > +# can directly load it into ram through tftp > > > +# > > > +# part - is a wic command that drive the process of generating a valid file > > system > > > +# - --source=uboot : wic plugin that generates a ramdisk image for U- > > Boot > > > +# - --fstype=ext2 : file system type( ext2 / ext3 / ext 4) > > > +# > > > +# %packages %end - option to provide a list of packages that will be > > installed > > > +# into rootfs. All packages dependencies will be installed by > > > +# package manager(default opkg). > > > + > > > + > > > +part / --source=uboot --fstype=ext2 --label imageName --align 1024 > > > + > > > +%packages > > > +%end > > > diff --git a/scripts/lib/image/config/wic.conf > > b/scripts/lib/image/config/wic.conf > > > index e96d6ae..2a2750b 100644 > > > --- a/scripts/lib/image/config/wic.conf > > > +++ b/scripts/lib/image/config/wic.conf > > > @@ -4,4 +4,8 @@ distro_name = OpenEmbedded > > > > > > [create] > > > ; settings for create subcommand > > > +; repourl=http://linux.com/ipk/all http://linux.com/ipk/target > > http://linux.com/ipk/arch > > > +arch=powerpc > > > +pkgmgr=opkg > > > runtime=native > > > +install_pkgs=source > > > > I don't think you want these settings as defaults in wic.conf. Maybe in > > the documentation? > [Adrian Calianu] What documentation? Please can you shade some light here? The documentation I'm referring to is the wic command-line documentation: $ wic help $ wic help create It's not very complete right now, and I have plans to expand it quite a bit - basically users should be able to get everything they need from that - but for now it would be ok to just add your content wherever it makes most sense and we can move it around later. The important thing is that it's there somewhere. > Sometimes from user perspective would be nice to have a fully > functional features without needed to read a manual to understand it > but here is difficult since settings are distribution dependent. > > > > > > diff --git a/scripts/lib/mic/imager/direct.py > > b/scripts/lib/mic/imager/direct.py > > > index 2cf4c8d..fef9d0e 100644 > > > --- a/scripts/lib/mic/imager/direct.py > > > +++ b/scripts/lib/mic/imager/direct.py > > > @@ -262,6 +262,10 @@ class DirectImageCreator(BaseImageCreator): > > > # when/if we need to actually do package selection we > > > # should modify things to use those objects, but for now > > > # we can avoid that. > > > + > > > + p.install_pkgs(self, self.workdir, self.oe_builddir, self.rootfs_dir, > > > + self.bootimg_dir, self.kernel_dir, self.native_sysroot) > > > + > > > p.prepare(self, self.workdir, self.oe_builddir, self.rootfs_dir, > > > self.bootimg_dir, self.kernel_dir, self.native_sysroot) > > > > > > diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py > > b/scripts/lib/mic/kickstart/custom_commands/partition.py > > > index 6b575c0..450d2d4 100644 > > > --- a/scripts/lib/mic/kickstart/custom_commands/partition.py > > > +++ b/scripts/lib/mic/kickstart/custom_commands/partition.py > > > @@ -31,7 +31,11 @@ from mic.utils.oe.misc import * > > > from mic.kickstart.custom_commands import * > > > from mic.plugin import pluginmgr > > > > > > +import os > > > +from mic.utils.oe.package_manager import * > > > + > > > partition_methods = { > > > + "do_install_pkgs":None, > > > "do_stage_partition":None, > > > "do_prepare_partition":None, > > > "do_configure_partition":None, > > > @@ -115,6 +119,102 @@ class Wic_PartData(Mic_PartData): > > > else: > > > return 0 > > > > > > + def install_pkgs(self, creator, cr_workdir, oe_builddir, rootfs_dir, > > > + bootimg_dir, kernel_dir, native_sysroot): > > > + """ > > > + Prepare content for individual partitions, installing packages. > > > + """ > > > + > > > + if not self.source: > > > + return > > > + > > > + self._source_methods = > > pluginmgr.get_source_plugin_methods(self.source, partition_methods) > > > + self._source_methods["do_install_pkgs"](self, creator, > > > + cr_workdir, > > > + oe_builddir, > > > + rootfs_dir, > > > + bootimg_dir, > > > + kernel_dir, > > > + native_sysroot) > > > + > > > + def install_pkgs_ipk(self, cr_workdir, oe_builddir, rootfs_dir, > > > + native_sysroot, packages, repourl): > > > + """ > > > + Install packages specified into wks file using opkg package manager. > > > + This method is dependend on bb module. > > > + """ > > > + > > > + gVar = {} > > > + gVar["DEPLOY_DIR_IPK"] = os.path.join(oe_builddir, > > "tmp/deploy/ipk") > > > + > > > + # Run postinstall scripts even in offline mode > > > + # Use the arch priority package rather than higher version one if more > > than one candidate is found. > > > + #d.setVar("OPKG_ARGS", "--force_postinstall --prefer-arch-to- > > version") > > > + gVar["OPKG_ARGS"] = "--force_postinstall" > > > + > > > + # OPKG path relative to /output_path > > > + gVar["OPKGLIBDIR"] = "var/lib" > > > + > > > + source_url = repourl.split() > > > + > > > + # Generate feed uri's names, it doesn't seem to matter what name > > they have > > > + feed_uris = "" > > > + cnt = 0 > > > + archs = "" > > > + for url in source_url: > > > + feed_uris += "cl_def_feed%d##%s\n" % (cnt, url) > > > + cnt += 1 > > > + head, tail = os.path.split(url) > > > + archs += " " + tail > > > + > > > + # IPK_FEED_URIS with special formating defines the URI's used as > > source for packages > > > + gVar['IPK_FEED_URIS'] = feed_uris > > > + > > > + gVar['BUILD_IMAGES_FROM_FEEDS'] = "1" > > > + > > > + # We need to provide sysroot for utilities > > > + gVar['STAGING_DIR_NATIVE'] = native_sysroot > > > + > > > + # Set WORKDIR for output > > > + gVar['WORKDIR'] = cr_workdir > > > + > > > + # Set TMPDIR for output > > > + gVar['TMPDIR'] = os.path.join(cr_workdir, "tmp") > > > + > > > + if 'ROOTFS_DIR' in rootfs_dir: > > > + target_dir = rootfs_dir['ROOTFS_DIR'] > > > + elif os.path.isdir(rootfs_dir): > > > + target_dir = rootfs_dir > > > + else: > > > + msg = "Couldn't find --rootfs-dir=%s connection" > > > + msg += " or it is not a valid path, exiting" > > > + msger.error(msg % rootfs_dir) > > > + > > > + # Need native sysroot /usr/bin/ for opkg-cl > > > + # chnage PATH var to avoid issues with host tools > > > + defpath = os.environ['PATH'] > > > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > > > + > > > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > > > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % > > target_dir > > > + pseudo += "export PSEUDO_PASSWD=%s;" % target_dir > > > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > > > + pseudo += "%s/usr/bin/pseudo " % native_sysroot > > > + > > > + pm = WicOpkgPM(gVar, > > > + target_dir, > > > + 'opkg.conf', > > > + archs, > > > + pseudo, > > > + native_sysroot) > > > + > > > + pm.update() > > > + > > > + pm.install(packages) > > > + > > > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > > > + > > > + > > > def prepare(self, cr, cr_workdir, oe_builddir, rootfs_dir, bootimg_dir, > > > kernel_dir, native_sysroot): > > > """ > > > @@ -225,6 +325,34 @@ class Wic_PartData(Mic_PartData): > > > > > > return 0 > > > > > > + def prepare_for_uboot(self, arch, cr_workdir, oe_builddir, rootfs_dir, > > > + native_sysroot): > > > + """ > > > + Generates u-boot image from source_file( ext2/3/4 ) > > > + > > > + """ > > > + pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > > > + pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % > > rootfs_dir > > > + pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir > > > + pseudo += "export PSEUDO_NOSYMLINKEXP=1;" > > > + pseudo += "%s/usr/bin/pseudo " % native_sysroot > > > + > > > + # 1) compress image > > > + rootfs = self.source_file > > > + rootfs_gzip = "%s.gz" % rootfs > > > + gzip_cmd = "gzip -f -9 -c %s > %s" % (rootfs, rootfs_gzip) > > > + rc, out = exec_native_cmd(pseudo + gzip_cmd, native_sysroot) > > > + > > > + # 2) image for U-Boot > > > + rootfs_uboot = "%s.u-boot" % rootfs_gzip > > > + mkimage_cmd = "mkimage -A %s -O linux -T ramdisk -C gzip -n %s -d > > %s %s" % \ > > > + (arch, self.label, rootfs_gzip, rootfs_uboot) > > > + rc, out = exec_native_cmd(pseudo + mkimage_cmd, native_sysroot) > > > + > > > + msger.info("\n\n\tThe new U-Boot ramdisk image can be found > > here:\n\t\t%s\n\n" % rootfs_uboot) > > > + > > > + return 0 > > > + > > > def prepare_rootfs_btrfs(self, cr_workdir, oe_builddir, rootfs_dir, > > > native_sysroot, pseudo): > > > """ > > > diff --git a/scripts/lib/mic/pluginbase.py b/scripts/lib/mic/pluginbase.py > > > index 9cf4c62..881d996 100644 > > > --- a/scripts/lib/mic/pluginbase.py > > > +++ b/scripts/lib/mic/pluginbase.py > > > @@ -89,6 +89,15 @@ class SourcePlugin(_Plugin): > > > """ > > > > > > @classmethod > > > + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, > > rootfs_dir, > > > + bootimg_dir, kernel_dir, native_sysroot): > > > + """ > > > + Called before partitions have been prepared and assembled into a > > > + disk image. Install packages into rootfs > > > + """ > > > + msger.debug("SourcePlugin: do_install_pkgs: part %s" % part) > > > + > > > + @classmethod > > > def do_install_disk(self, disk, disk_name, cr, workdir, oe_builddir, > > > bootimg_dir, kernel_dir, native_sysroot): > > > """ > > > diff --git a/scripts/lib/mic/plugins/source/uboot.py > > b/scripts/lib/mic/plugins/source/uboot.py > > > new file mode 100644 > > > index 0000000..57cb3cf > > > --- /dev/null > > > +++ b/scripts/lib/mic/plugins/source/uboot.py > > > @@ -0,0 +1,173 @@ > > > +# ex:ts=4:sw=4:sts=4:et > > > +# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > > > +# > > > +# Copyright (c) 2014, Enea AB. > > > +# All rights reserved. > > > +# > > > +# This program is free software; you can redistribute it and/or modify > > > +# it under the terms of the GNU General Public License version 2 as > > > +# published by the Free Software Foundation. > > > +# > > > +# This program is distributed in the hope that it will be useful, > > > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > > > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > > > +# GNU General Public License for more details. > > > +# > > > +# You should have received a copy of the GNU General Public License > > along > > > +# with this program; if not, write to the Free Software Foundation, Inc., > > > +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > > > +# > > > +# DESCRIPTION > > > +# This implements the 'uboot' source plugin class for 'wic' > > > +# > > > +# AUTHORS > > > +# Adrian Calianu <adrian.calianu (at] enea.com> > > > +# > > > + > > > +import os > > > +import shutil > > > +import re > > > +import tempfile > > > + > > > +from mic import kickstart, chroot, msger > > > +from mic.utils import misc, fs_related, errors, runner, cmdln > > > +from mic.conf import configmgr > > > +from mic.plugin import pluginmgr > > > +from mic.utils.partitionedfs import PartitionedMount > > > +import mic.imager.direct as direct > > > +from mic.pluginbase import SourcePlugin > > > +from mic.utils.oe.misc import * > > > +from mic.imager.direct import DirectImageCreator > > > + > > > +def create_local_rootfs(part, creator, cr_workdir, krootfs_dir, > > native_sysroot): > > > + # In order to have a full control over rootfs we will make a local copy > > under workdir > > > + # and change rootfs_dir to new location. > > > + # In this way we can install more than one ROOTFS_DIRs and/or use > > > + # an empty rootfs to install packages, so a rootfs could be generated > > only from pkgs > > > + # TBD: create workdir/rootfs ; copy rootfs-> workdir/rootfs; set > > rootfs=workdir/rootfs > > > + > > > > Is this something that would be needed by any image creating a rootfs > > from packages? If so, it should be part of the packaging functionality, > > not the uboot implementation specifically. > [Adrian Calianu] > Yes, indeed it can be moved outside of uboot plugin. > > > > > > + cr_workdir = os.path.abspath(cr_workdir) > > > + new_rootfs_dir = "%s/rootfs_%s" % (cr_workdir, creator.name) > > > + > > > + rootfs_exists = 1 > > > + if part.rootfs is None: > > > + if not 'ROOTFS_DIR' in krootfs_dir: > > > + msg = "Couldn't find --rootfs-dir, exiting, " > > > + msger.info(msg) > > > + rootfs_exists = 0 > > > + rootfs_dir = krootfs_dir['ROOTFS_DIR'] > > > + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir > > > + else: > > > + if part.rootfs in krootfs_dir: > > > + rootfs_dir = krootfs_dir[part.rootfs] > > > + creator.rootfs_dir[part.rootfs] = new_rootfs_dir > > > + elif os.path.isdir(part.rootfs): > > > + rootfs_dir = part.rootfs > > > + part.rootfs = new_rootfs_dir > > > + else: > > > + msg = "Couldn't find --rootfs-dir=%s connection" > > > + msg += " or it is not a valid path, exiting" > > > + msger.info(msg % part.rootfs) > > > + rootfs_exists = 0 > > > + creator.rootfs_dir['ROOTFS_DIR'] = new_rootfs_dir > > > + > > > + pseudox = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > > > + pseudox += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % > > new_rootfs_dir > > > + pseudox += "export PSEUDO_PASSWD=%s;" % new_rootfs_dir > > > + pseudox += "export PSEUDO_NOSYMLINKEXP=1;" > > > + pseudox += "%s/usr/bin/pseudo " % native_sysroot > > > + > > > + mkdir_cmd = "mkdir %s" % (new_rootfs_dir) > > > + # rc, out = exec_native_cmd(pseudox + mkdir_cmd, native_sysroot) > > > + rc, out = exec_cmd(mkdir_cmd, True) > > > + > > > + if rootfs_exists == 1 and os.path.isdir(rootfs_dir): > > > + defpath = os.environ['PATH'] > > > + os.environ['PATH'] = native_sysroot + "/usr/bin/" + ":/bin:/usr/bin:" > > > + > > > + rootfs_dir = os.path.abspath(rootfs_dir) > > > + > > > + pseudoc = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot > > > + pseudoc += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % > > rootfs_dir > > > + pseudoc += "export PSEUDO_PASSWD=%s;" % rootfs_dir > > > + pseudoc += "export PSEUDO_NOSYMLINKEXP=1;" > > > + pseudoc += "%s/usr/bin/pseudo " % native_sysroot > > > + > > > + tarc_cmd = "tar cvpf %s/rootfs.tar -C %s ." % (cr_workdir, rootfs_dir) > > > + rc, out = exec_native_cmd(pseudoc + tarc_cmd, native_sysroot) > > > + > > > + tarx_cmd = "tar xpvf %s/rootfs.tar -C %s" % (cr_workdir, > > new_rootfs_dir) > > > + rc, out = exec_native_cmd(pseudox + tarx_cmd, native_sysroot) > > > + > > > + rm_cmd = "rm %s/rootfs.tar" % cr_workdir > > > + rc, out = exec_cmd(rm_cmd, True) > > > + > > > + os.environ['PATH'] += defpath + ":" + native_sysroot + "/usr/bin/" > > > + > > > + return new_rootfs_dir > > > + > > > +class UBootPlugin(SourcePlugin): > > > + name = 'uboot' > > > + > > > + @classmethod > > > + def do_install_pkgs(self, part, creator, cr_workdir, oe_builddir, > > krootfs_dir, > > > + bootimg_dir, kernel_dir, native_sysroot): > > > + """ > > > + Called before all partitions have been prepared and assembled into a > > > + disk image. Intall packages based on wic configuration. > > > + """ > > > + > > > > Same here, this looks like general-purpose package installation code > > that shouldn't be part of a u-boot plugin. Also, is that comment > > correct? > [Adrian Calianu] This method is part of a more generic pluginbase > class and each source plugin has the option to generate rootfs from > packages or not. > It could happen to have more partitions and each partition with a > specific rootfs. That specific rootfs can be generated (installing > packages) or user specified (--rootfs) and this is controlled by > source plugin. > Right, so this code could be moved into a more generic plugin - it isn't uboot-specific, right? Tom [snip] ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <53996893.88fd440a.0627.25d6SMTPIN_ADDED_BROKEN@mx.google.com>]
* Re: [PATCHv2 1/2] Add wic support to generate rootfs image for uboot [not found] ` <53996893.88fd440a.0627.25d6SMTPIN_ADDED_BROKEN@mx.google.com> @ 2014-06-12 14:02 ` Otavio Salvador 2014-06-13 6:43 ` Adrian Calianu 0 siblings, 1 reply; 11+ messages in thread From: Otavio Salvador @ 2014-06-12 14:02 UTC (permalink / raw) To: y; +Cc: Patches and discussions about the oe-core layer Hello, On Thu, Jun 12, 2014 at 5:43 AM, <y@enea.se> wrote: > From: Adrian Calianu <adrian.calianu@enea.com> > > Add support in wic to build ramdisk uboot images from list of packages or > from an existing rootfs folder. > > Providing the list of packages starting with XXX-core-boot package(into wks file) > will generate an rootfs image for uboot out of bitbake. > There is also possible to combine an existing/application rootfs with a list > of packages that can be installed but the user must take care to provide > a basic core boot rootfs or XXX-core-boot package and reporul(wic.conf). > > Some prerequisites are required for this new build setup: > ../poky/scripts/lib/image/config/wic.conf > [create] > arch=target_arch (example: powerpc, arm, x86) > pkgmgr=opkg > repourl=http://example.distro/p2020rdb/ipk/all http://example.distro/p2020rdb/ipk/p2020rdb http://example.distro/p2020rdb/ipk/ppce500v2 > > 1) Build an rootfs image from an existing bitbake build: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e core-image-minimal -o .../output > > 2) Build an rootfs image from an existing rootfs and native_sysroot: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r .../path_to_rootfs -n .../path_to_native_sysroot -o .../output > > 3) Build an rootfs image only from a package list (on wks file): > > Add the package list to be installed as rootfs on ../poky/scripts/lib/image/canned-wks/uboot.wks: > %packages > packagegroup-core-boot > pramfs-init > run-postinsts > packagegroup-core-ssh-dropbear > %end > > Generate rootfs image: > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n .../path_to_native_sysroot -o .../output > > Signed-off-by: Adrian Calianu <adrian.calianu@enea.com> I really liked the idea and it is a nice step further but it is not clear to me how we'll "tell" wic about platform specific U-Boot installation like i.MX vs MXS where the layout of the uSD image is very different. -- Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv2 1/2] Add wic support to generate rootfs image for uboot 2014-06-12 14:02 ` Otavio Salvador @ 2014-06-13 6:43 ` Adrian Calianu 0 siblings, 0 replies; 11+ messages in thread From: Adrian Calianu @ 2014-06-13 6:43 UTC (permalink / raw) To: Otavio Salvador, Adrian Calianu Cc: Patches and discussions about the oe-core layer Hi Otavio, First step was to add support in wic to generate rootfs images only from packages out of bitbake environment. To test this functionality was chosen to generate ramdisk images for now. We have plans to extend the U-Boot plugin in order to generate other types of rootfs images. Thanks, Adrian > -----Original Message----- > From: openembedded-core-bounces@lists.openembedded.org > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf > Of Otavio Salvador > Sent: Thursday, June 12, 2014 5:02 PM > To: y@enea.se > Cc: Patches and discussions about the oe-core layer > Subject: Re: [OE-core] [PATCHv2 1/2] Add wic support to generate rootfs > image for uboot > > Hello, > > On Thu, Jun 12, 2014 at 5:43 AM, <y@enea.se> wrote: > > From: Adrian Calianu <adrian.calianu@enea.com> > > > > Add support in wic to build ramdisk uboot images from list of packages > > or from an existing rootfs folder. > > > > Providing the list of packages starting with XXX-core-boot > > package(into wks file) will generate an rootfs image for uboot out of > bitbake. > > There is also possible to combine an existing/application rootfs with > > a list of packages that can be installed but the user must take care > > to provide a basic core boot rootfs or XXX-core-boot package and > reporul(wic.conf). > > > > Some prerequisites are required for this new build setup: > > ../poky/scripts/lib/image/config/wic.conf > > [create] > > arch=target_arch (example: powerpc, arm, x86) pkgmgr=opkg > > repourl=http://example.distro/p2020rdb/ipk/all > > http://example.distro/p2020rdb/ipk/p2020rdb > > http://example.distro/p2020rdb/ipk/ppce500v2 > > > > 1) Build an rootfs image from an existing bitbake build: > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -e > > core-image-minimal -o .../output > > > > 2) Build an rootfs image from an existing rootfs and native_sysroot: > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -r > > .../path_to_rootfs -n .../path_to_native_sysroot -o .../output > > > > 3) Build an rootfs image only from a package list (on wks file): > > > > Add the package list to be installed as rootfs on > ../poky/scripts/lib/image/canned-wks/uboot.wks: > > %packages > > packagegroup-core-boot > > pramfs-init > > run-postinsts > > packagegroup-core-ssh-dropbear > > %end > > > > Generate rootfs image: > > wic create ../poky/scripts/lib/image/canned-wks/uboot.wks -n > > .../path_to_native_sysroot -o .../output > > > > Signed-off-by: Adrian Calianu <adrian.calianu@enea.com> > > I really liked the idea and it is a nice step further but it is not clear to me how > we'll "tell" wic about platform specific U-Boot installation like i.MX vs MXS > where the layout of the uSD image is very different. > > -- > Otavio Salvador O.S. Systems > http://www.ossystems.com.br http://code.ossystems.com.br > Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750 > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-06-20 14:46 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-06-10 13:41 [PATCH 0/1] wic: Add wic support to generate rootfs image for uboot Adrian Calianu
2014-06-10 13:41 ` [PATCH 1/1] " Adrian Calianu
2014-06-12 2:49 ` Saul Wold
2014-06-12 8:43 ` [PATCHv2 0/2] wic: " y
2014-06-12 8:43 ` [PATCHv2 1/2] " y
2014-06-13 20:18 ` Tom Zanussi
2014-06-16 13:18 ` Adrian Calianu
2014-06-17 9:35 ` Anders Darander
2014-06-20 14:46 ` Tom Zanussi
[not found] ` <53996893.88fd440a.0627.25d6SMTPIN_ADDED_BROKEN@mx.google.com>
2014-06-12 14:02 ` Otavio Salvador
2014-06-13 6:43 ` Adrian Calianu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.