[PATCH 1/4] image_types_wic.bbclass: Copy the .wks and .env files to deploy image dir

[PATCH 1/4] image_types_wic.bbclass: Copy the .wks and .env files to deploy image dir

[Jason Wessel]

When using a .wks.in file, the only place that the generated .wks file
exists in the tmp/work area.  A copy should be left behind in the
deploy directory so that you can easily run the wic tool to
re-generate or modify a new image without running bitbake.  Custom
.wks.in files can reference any number of bitbake variables, so it is
important to save the result.

below is an example of using the generated .wks file in the deploy
area.  The full name of my generated .wks file was
core-image-minimal-ostree-uboot-ab.wks, but since you usually only
have a single .wks file per image you can use a wild card like:

  cd tmp*/deploy/images/*
  wic create --vars . -e core-image-minimal -s -m core-image-minimal-*.wks

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/classes/image_types_wic.bbclass | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index 519aeb1b04..ce94138b3c 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -77,6 +77,11 @@ python do_write_wks_template () {
     wks_file = d.getVar('WKS_FULL_PATH')
     with open(wks_file, 'w') as f:
         f.write(template_body)
+    f.close()
+    # Copy the finalized wks file to the deploy directory for later use
+    depdir = d.getVar('DEPLOY_DIR_IMAGE')
+    basename = d.getVar('IMAGE_BASENAME')
+    bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + os.path.basename(wks_file)))
 }
 
 python () {
@@ -127,6 +132,10 @@ python do_rootfs_wicenv () {
             value = d.getVar(var)
             if value:
                 envf.write('%s="%s"\n' % (var, value.strip()))
+    envf.close()
+    # Copy to deploy directory for later use with stand alone wic
+    depdir = d.getVar('DEPLOY_DIR_IMAGE')
+    bb.utils.copyfile(os.path.join(outdir, basename) + '.env', os.path.join(depdir, basename) + '.env')
 }
 addtask do_rootfs_wicenv after do_image before do_image_wic
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
-- 
2.21.0

[PATCH 2/4] wic: Add partition type for msdos partition tables

[Jason Wessel]

In order to create a msdos partition table disk image that can auto
expand after the image is copied to an SD card, wic needs the ability
to have a primary partition as the last entry.  The desired use case
is to be able to create an A/B update partition image scheme with a
/var volume that can be auto expanded to the remainder of the SD card
at run time.

The typical .wks file will look similar to the following:

bootloader --ptable msdos
part / --source rawcopy --sourceparams="file=u-boot.imx" \
   --ondisk mmcblk --no-table --align 1 --size 1
part /boot --source bootimg-partition \
   --ondisk mmcblk --fstype=vfat --label boot --active --align 4 --size 16
part / --source rawcopy --sourceparams="file=imx6_boot.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaboot --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaroot --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6_boot.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaboot_b --align 4 --type logical
part / --source rawcopy --sourceparams="file=imx6.otaimg" \
   --ondisk mmcblk --fstype=ext4 --label otaroot_b --align 4 --type logical
part /var --source rawcopy --sourceparams="file=imx6_var.otaimg" \
    --ondisk mmcblk --fstype=ext4 --label fluxdata --align 4

Without the patch applied, wic will generate an SD card image that looks like:

Disk boot.img: 890940s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start    End      Size     Type      File system  Flags
 1      2056s    48001s   45946s   primary   fat16        lba
 2      48008s   132467s  84460s   primary   ext4
 3      132472s  454467s  321996s  primary   ext4
 4      454471s  890939s  436469s  extended               lba
 5      454472s  538931s  84460s   logical   ext4
 6      538936s  860931s  321996s  logical   ext4
 7      860936s  890939s  30004s   logical   ext4         boot

With the patch applied a primary partition can be created at the end
of the image which can be expanded to fill the free space on the media
where the image has been copied, which looks like:

Disk boot.img: 890940s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start    End      Size     Type      File system  Flags
 1      2056s    48001s   45946s   primary   fat16        lba
 2      48007s   860931s  812925s  extended               lba
 5      48008s   132467s  84460s   logical   ext4
 6      132472s  454467s  321996s  logical   ext4
 7      454472s  538931s  84460s   logical   ext4
 8      538936s  860931s  321996s  logical   ext4
 3      860936s  890939s  30004s   primary   ext4         boot

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 scripts/lib/wic/ksparser.py              |  2 ++
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/imager/direct.py | 37 +++++++++++++++++-------
 3 files changed, 29 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 3e670033ba..6a643ba3af 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -151,6 +151,8 @@ class KickStart():
         part.add_argument('--part-name')
         part.add_argument('--part-type')
         part.add_argument('--rootfs-dir')
+        part.add_argument('--type', default='primary',
+                choices = ('primary', 'logical'))
 
         # --size and --fixed-size cannot be specified together; options
         # ----extra-space and --overhead-factor should also raise a parser
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 01466b258d..f427c8101b 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -50,6 +50,7 @@ class Partition():
         self.use_uuid = args.use_uuid
         self.uuid = args.uuid
         self.fsuuid = args.fsuuid
+        self.type = args.type
 
         self.lineno = lineno
         self.source_file = ""
diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/imager/direct.py
index 91fc5e70f6..3ce6ad55b8 100644
--- a/scripts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/imager/direct.py
@@ -300,6 +300,10 @@ class PartitionedImage():
         self.path = path  # Path to the image file
         self.numpart = 0  # Number of allocated partitions
         self.realpart = 0 # Number of partitions in the partition table
+        self.primary_part_num = 0  # Number of primary partitions (msdos)
+        self.extendedpart = 0      # Create extended partition before this logical partition (msdos)
+        self.extended_size_sec = 0 # Size of exteded partition (msdos)
+        self.logical_part_cnt = 0  # Number of total logical paritions (msdos)
         self.offset = 0   # Offset of next partition (in sectors)
         self.min_size = 0 # Minimum required disk size to fit
                           # all partitions (in bytes)
@@ -391,12 +395,16 @@ class PartitionedImage():
                 # Skip one sector required for the partitioning scheme overhead
                 self.offset += overhead
 
-            if self.realpart > 3 and num_real_partitions > 4:
+            if self.ptable_format == "msdos":
+                if self.primary_part_num > 3 or \
+                   (self.extendedpart == 0 and self.primary_part_num >= 3 and num_real_partitions > 4):
+                    part.type = 'logical'
                 # Reserve a sector for EBR for every logical partition
                 # before alignment is performed.
-                if self.ptable_format == "msdos":
+                if part.type == 'logical':
                     self.offset += 1
 
+            align_sectors = 0
             if part.align:
                 # If not first partition and we do have alignment set we need
                 # to align the partition.
@@ -422,18 +430,25 @@ class PartitionedImage():
             part.start = self.offset
             self.offset += part.size_sec
 
-            part.type = 'primary'
             if not part.no_table:
                 part.num = self.realpart
             else:
                 part.num = 0
 
-            if self.ptable_format == "msdos":
-                # only count the partitions that are in partition table
-                if num_real_partitions > 4:
-                    if self.realpart > 3:
-                        part.type = 'logical'
-                        part.num = self.realpart + 1
+            if self.ptable_format == "msdos" and not part.no_table:
+                if part.type == 'logical':
+                    self.logical_part_cnt += 1
+                    part.num = self.logical_part_cnt + 4
+                    if self.extendedpart == 0:
+                        # Create extended partition as a primary partition
+                        self.primary_part_num += 1
+                        self.extendedpart = part.num
+                    else:
+                        self.extended_size_sec += align_sectors
+                    self.extended_size_sec += part.size_sec + 1
+                else:
+                    self.primary_part_num += 1
+                    part.num = self.primary_part_num
 
             logger.debug("Assigned %s to %s%d, sectors range %d-%d size %d "
                          "sectors (%d bytes).", part.mountpoint, part.disk,
@@ -483,7 +498,7 @@ class PartitionedImage():
             if part.num == 0:
                 continue
 
-            if self.ptable_format == "msdos" and part.num == 5:
+            if self.ptable_format == "msdos" and part.num == self.extendedpart:
                 # Create an extended partition (note: extended
                 # partition is described in MBR and contains all
                 # logical partitions). The logical partitions save a
@@ -497,7 +512,7 @@ class PartitionedImage():
                 # room for all logical partitions.
                 self._create_partition(self.path, "extended",
                                        None, part.start - 1,
-                                       self.offset - part.start + 1)
+                                       self.extended_size_sec)
 
             if part.fstype == "swap":
                 parted_fs_type = "linux-swap"
-- 
2.21.0

[PATCH 3/4] wic: Make disk partition size consistently computed

[Jason Wessel]

When using different root directories with a wks file wic is using the
value from the original ROOTFS_SIZE which is not correct.  Example:

Number  Start   End     Size    File system  Name       Flags
 1      20.5kB  318MB   318MB   fat16        otaefi     legacy_boot, msftdata
 2      318MB   636MB   318MB   ext4         otaboot
 3      636MB   1709MB  1074MB  ext4         otaroot
 4      1709MB  2027MB  318MB   ext4         otaboot_b
 5      2027MB  3101MB  1074MB  ext4         otaroot_b
 6      3101MB  5249MB  2147MB  ext4         fluxdata

The partitions 1, 2, and 3 incorrectly inherit the size, instead of using
a computed size.  With the patch applied it is working properly:

Number  Start   End     Size    File system  Name       Flags
 1      20.5kB  14.5MB  14.5MB  fat16        otaefi     legacy_boot, msftdata
 2      14.5MB  65.3MB  50.8MB  ext4         otaboot
 3      65.3MB  1139MB  1074MB  ext4         otaroot
 4      1139MB  1190MB  50.8MB  ext4         otaboot_b
 5      1190MB  2264MB  1074MB  ext4         otaroot_b
 6      2264MB  4411MB  2147MB  ext4         fluxdata

As for the removal of the bb.warn, if the size is not specified, it is
not something to warn the end user about.  Some of my default images
make use of the head room + a computed directory size or while generating
images.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 scripts/lib/wic/partition.py | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index f427c8101b..2a71d7b1d6 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -212,19 +212,13 @@ class Partition():
         if os.path.isfile(rootfs):
             os.remove(rootfs)
 
-        # Get rootfs size from bitbake variable if it's not set in .ks file
+        # If size is not specified compute it from the rootfs_dir size
         if not self.size and real_rootfs:
-            # Bitbake variable ROOTFS_SIZE is calculated in
-            # Image._get_rootfs_size method from meta/lib/oe/image.py
-            # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
-            # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
-            rsize_bb = get_bitbake_var('ROOTFS_SIZE')
-            if rsize_bb:
-                logger.warning('overhead-factor was specified, but size was not,'
-                               ' so bitbake variables will be used for the size.'
-                               ' In this case both IMAGE_OVERHEAD_FACTOR and '
-                               '--overhead-factor will be applied')
-                self.size = int(round(float(rsize_bb)))
+            # Use the same logic found in get_rootfs_size()
+            # from meta/classes/image.bbclass
+            du_cmd = "du -ks %s" % rootfs_dir
+            out = exec_cmd(du_cmd)
+            self.size = int(out.split()[0])
 
         prefix = "ext" if self.fstype.startswith("ext") else self.fstype
         method = getattr(self, "prepare_rootfs_" + prefix)
-- 
2.21.0

[PATCH 4/4] dpkg: Provide update-alternative for start-stop-daemon

[Jason Wessel]

When using DISTRO_FEATURE usrmerge, busybox and something that
requires dpkg-start-stop there is a conflict which prints the
following error.

===

ERROR: core-image-minimal-1.0-r1 do_rootfs: Postinstall scriptlets of ['busybox'] have failed. If the intention is to defer them to first boot,
then please place them into pkg_postinst_ontarget_${PN} ().
Deferring to first boot via 'exit 1' is no longer supported.
Details of the failure are in /build/tmp-glibc/work/intel_x86_64-linux/core-image-minimal/1.0-r1/temp/log.do_rootfs.
ERROR: core-image-minimal-1.0-r1 do_rootfs: Function failed: do_rootfs

===

Looking at the log file, it shows the problem:

update-alternatives: Error: not linking /build/tmp-glibc/work/intel_x86_64-linux/core-image-minimal/1.0-r1/rootfs/usr/sbin/start-stop-daemon to /usr/bin/busybox.nosuid since /build/tmp-glibc/work/intel_x86_64-linux/core-image-minimal/1.0-r1/rootfs/usr/sbin/start-stop-daemon exists and is not a link

===

The solution is to add an update-alternative to start-stop-daemon for dpkg.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/recipes-devtools/dpkg/dpkg.inc | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/dpkg/dpkg.inc b/meta/recipes-devtools/dpkg/dpkg.inc
index fee7a75814..1c3c585d79 100644
--- a/meta/recipes-devtools/dpkg/dpkg.inc
+++ b/meta/recipes-devtools/dpkg/dpkg.inc
@@ -9,7 +9,7 @@ RDEPENDS_${PN}_class-native = ""
 
 UPSTREAM_CHECK_URI = "${DEBIAN_MIRROR}/main/d/dpkg/"
 
-inherit autotools gettext perlnative pkgconfig perl-version
+inherit autotools gettext perlnative pkgconfig perl-version update-alternatives
 
 PERL_class-native = "${STAGING_BINDIR_NATIVE}/perl-native/perl"
 
@@ -76,7 +76,10 @@ RDEPENDS_${PN}-perl += "perl-module-carp perl-module-constant \
 # the bitbake version.
 #
 PACKAGES =+ "${PN}-start-stop"
-FILES_${PN}-start-stop = "${sbindir}/start-stop-daemon"
+FILES_${PN}-start-stop = "${sbindir}/start-stop-daemon.${BPN}"
+ALTERNATIVE_${PN}-start-stop = "start-stop-daemon"
+ALTERNATIVE_LINK_NAME[start-stop-daemon] = "${sbindir}/start-stop-daemon"
+ALTERNATIVE_PRIORITY = "100"
 
 RDEPENDS_${PN} += "${PN}-start-stop"
 
-- 
2.21.0

Re: [PATCH 1/4] image_types_wic.bbclass: Copy the .wks and .env files to deploy image dir

[Jason Wessel]

On 8/8/19 7:55 AM, Jason Wessel wrote:
> When using a .wks.in file, the only place that the generated .wks file
> exists in the tmp/work area.  A copy should be left behind in the
> deploy directory so that you can easily run the wic tool to
> re-generate or modify a new image without running bitbake.  Custom
> .wks.in files can reference any number of bitbake variables, so it is
> important to save the result.
> 
> below is an example of using the generated .wks file in the deploy
> area.  The full name of my generated .wks file was
> core-image-minimal-ostree-uboot-ab.wks, but since you usually only
> have a single .wks file per image you can use a wild card like:
> 
>    cd tmp*/deploy/images/*
>    wic create --vars . -e core-image-minimal -s -m core-image-minimal-*.wks
> 
> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> ---
>   meta/classes/image_types_wic.bbclass | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
> index 519aeb1b04..ce94138b3c 100644
> --- a/meta/classes/image_types_wic.bbclass
> +++ b/meta/classes/image_types_wic.bbclass
> @@ -77,6 +77,11 @@ python do_write_wks_template () {
>       wks_file = d.getVar('WKS_FULL_PATH')
>       with open(wks_file, 'w') as f:
>           f.write(template_body)
> +    f.close()
> +    # Copy the finalized wks file to the deploy directory for later use
> +    depdir = d.getVar('DEPLOY_DIR_IMAGE')

This needs a call to "bb.utils.mkdirhier(depdir)" in order to work properly.

I didn't find the problem until using a higher degree of parallelism where the deploy directory might not actually be there to write out the file.  I'll send a v2 after more tests with the rest of the series.  It was all well tested originally on a single server, extended testing found the problem.


Cheers,
Jason.

> +    basename = d.getVar('IMAGE_BASENAME')
> +    bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + os.path.basename(wks_file)))
>   }
>   
>   python () {
> @@ -127,6 +132,10 @@ python do_rootfs_wicenv () {
>               value = d.getVar(var)
>               if value:
>                   envf.write('%s="%s"\n' % (var, value.strip()))
> +    envf.close()
> +    # Copy to deploy directory for later use with stand alone wic
> +    depdir = d.getVar('DEPLOY_DIR_IMAGE')
> +    bb.utils.copyfile(os.path.join(outdir, basename) + '.env', os.path.join(depdir, basename) + '.env')
>   }
>   addtask do_rootfs_wicenv after do_image before do_image_wic
>   do_rootfs_wicenv[vardeps] += "${WICVARS}"
> 

Re: [PATCH 1/4] image_types_wic.bbclass: Copy the .wks and .env files to deploy image dir

[Richard Purdie]

On Thu, 2019-08-08 at 05:55 -0700, Jason Wessel wrote:
> When using a .wks.in file, the only place that the generated .wks file
> exists in the tmp/work area.  A copy should be left behind in the
> deploy directory so that you can easily run the wic tool to
> re-generate or modify a new image without running bitbake.  Custom
> .wks.in files can reference any number of bitbake variables, so it is
> important to save the result.
> 
> below is an example of using the generated .wks file in the deploy
> area.  The full name of my generated .wks file was
> core-image-minimal-ostree-uboot-ab.wks, but since you usually only
> have a single .wks file per image you can use a wild card like:
> 
>   cd tmp*/deploy/images/*
>   wic create --vars . -e core-image-minimal -s -m core-image-minimal-*.wks
> 
> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> ---
>  meta/classes/image_types_wic.bbclass | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
> index 519aeb1b04..ce94138b3c 100644
> --- a/meta/classes/image_types_wic.bbclass
> +++ b/meta/classes/image_types_wic.bbclass
> @@ -77,6 +77,11 @@ python do_write_wks_template () {
>      wks_file = d.getVar('WKS_FULL_PATH')
>      with open(wks_file, 'w') as f:
>          f.write(template_body)
> +    f.close()
> +    # Copy the finalized wks file to the deploy directory for later use
> +    depdir = d.getVar('DEPLOY_DIR_IMAGE')
> +    basename = d.getVar('IMAGE_BASENAME')
> +    bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + os.path.basename(wks_file)))
>  }

Nothing should be writing to DEPLOY_DIR_IMAGE directly as it means its
not being managed by sstate and won't restore in a build from sstate.
There is an intermediate directory it should be written to and the
sstate code will put it into DEPLOY_DIR_IMAGE. For image classes it
might be IMGDEPLOYDIR but you'd have to check this task/code runs
between do_image and do_image_complete.

>  python () {
> @@ -127,6 +132,10 @@ python do_rootfs_wicenv () {
>              value = d.getVar(var)
>              if value:
>                  envf.write('%s="%s"\n' % (var, value.strip()))
> +    envf.close()
> +    # Copy to deploy directory for later use with stand alone wic
> +    depdir = d.getVar('DEPLOY_DIR_IMAGE')
> +    bb.utils.copyfile(os.path.join(outdir, basename) + '.env', os.path.join(depdir, basename) + '.env')
>  }

Same here.

Cheers,

Richard

Re: [PATCH 1/4] image_types_wic.bbclass: Copy the .wks and .env files to deploy image dir

[Jason Wessel]

On 8/8/19 1:42 PM, Richard Purdie wrote:
> On Thu, 2019-08-08 at 05:55 -0700, Jason Wessel wrote:
>> When using a .wks.in file, the only place that the generated .wks file
>> exists in the tmp/work area.  A copy should be left behind in the
>> deploy directory so that you can easily run the wic tool to
>> re-generate or modify a new image without running bitbake.  Custom
>> .wks.in files can reference any number of bitbake variables, so it is
>> important to save the result.
>>
>> below is an example of using the generated .wks file in the deploy
>> area.  The full name of my generated .wks file was
>> core-image-minimal-ostree-uboot-ab.wks, but since you usually only
>> have a single .wks file per image you can use a wild card like:
>>
>>    cd tmp*/deploy/images/*
>>    wic create --vars . -e core-image-minimal -s -m core-image-minimal-*.wks
>>
>> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
>> ---
>>   meta/classes/image_types_wic.bbclass | 9 +++++++++
>>   1 file changed, 9 insertions(+)
>>
>> diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
>> index 519aeb1b04..ce94138b3c 100644
>> --- a/meta/classes/image_types_wic.bbclass
>> +++ b/meta/classes/image_types_wic.bbclass
>> @@ -77,6 +77,11 @@ python do_write_wks_template () {
>>       wks_file = d.getVar('WKS_FULL_PATH')
>>       with open(wks_file, 'w') as f:
>>           f.write(template_body)
>> +    f.close()
>> +    # Copy the finalized wks file to the deploy directory for later use
>> +    depdir = d.getVar('DEPLOY_DIR_IMAGE')
>> +    basename = d.getVar('IMAGE_BASENAME')
>> +    bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + os.path.basename(wks_file)))
>>   }
> 
> Nothing should be writing to DEPLOY_DIR_IMAGE directly as it means its
> not being managed by sstate and won't restore in a build from sstate.
> There is an intermediate directory it should be written to and the
> sstate code will put it into DEPLOY_DIR_IMAGE. For image classes it
> might be IMGDEPLOYDIR but you'd have to check this task/code runs
> between do_image and do_image_complete.
> 


Thank you for the pointer.  It works exactly as intended so long as the dependency
is changed properly, and the correct staging directory is used.
There is no directory creation required because the do_image() takes care of all
that.

I'll send v2 of the series, but this this the only patch that changed.

Jason.