From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2615CC4725D for ; Fri, 19 Jan 2024 08:38:36 +0000 (UTC) Subject: [PATCH] [WIC] Configurable addressing of GTP main table To: openembedded-core@lists.openembedded.org From: davide.cardillo@seco.com X-Originating-Location: Florence, Tuscany, IT (94.89.17.99) X-Originating-Platform: Linux Chrome 120 User-Agent: GROUPS.IO Web Poster MIME-Version: 1.0 Date: Fri, 19 Jan 2024 00:38:29 -0800 Message-ID: Content-Type: multipart/alternative; boundary="TrCFxH6zZF2drAJkL235" List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Fri, 19 Jan 2024 08:38:36 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/194018 --TrCFxH6zZF2drAJkL235 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Issue: the usage of a gpt table with i.MX processor is not compatible by default. This is due to a memory conflict: the main partition table is placed into the same memory space used to flash the header of the U-Boot. This last can not be moved since the ROM code of the i.MX6 starts to read to that fixed address (0x400). This problem can be solved using a feature of the GTP partition table for witch the main table can be moved to a desired memory area. The WIC library present in Poky manages the GTP table through some standard Linux commands (gdisk and sgdisk) which ones have already the opportunity to perform this task. This patch introduces this feature with the addition of some wic's options, used also in wks file. New partition placement behavior: Since now all partitions are placed after the gpt table presents into the first 1KB of memory (including both the gpt header at the first byte of memory and the main partition table starting at 0x400). Now the main gpt table can be placed in other address spaces and a partitio= n can be placed also before the gpt table. Recap: - by default, all partitions are placed sequentially, one after the other. The first partition implicitly starts after the GPT partition table. - adding a flag to the item of a partition into the wks file, the partition can be placed, starting from a fixed memory offset (neglecting the memory space occupied by the GTP table). Combining the moving of the main partition table and the absolute (not yet relative) positioning of a partition, now it is possible to place one or mo= re partitions before the GPT table. New flags: - main-ptable-offset (size in KB): to add as property of the bootloader ite= m. Specifies the main partition table location in KB. - fixed-align (boolean): to add as property of a partition. Specifies that = the partition must to placed with absolute offset, used in conjunction with --a= lign property Example: setting used for a wic for an i.MX6 processor, where the main partition tab= le is placed with the absolute offset of 4MB bootloader --ptable gpt --main-ptable-offset 4096K part / --source rawcopy --sourceparams=3D"file=3Du-boot.imx" --ondisk mmcbl= k --align 1 --size 1M --no-table --fixed-align Test environment: This patch is a part of Edgehog OS (https://git.seco.com/edgehog) a custom Yocto distribution by SECO SpA (www.seco.com), used on multiple architectures (both ARM and x86). This solution is widely used by all SECO customers who use hardware solution based on NXP i.MX6 processors. Signed-off-by: Davide Cardillo Reviewed-by: Tobias Poganiuch Tested-by: Andrea Da Col Tested-by: Daniel Rinaldi --- scripts/lib/wic/ksparser.py =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| =C2=A0=C2=A02 + scripts/lib/wic/partition.py =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0| =C2=A0=C2=A01 + scripts/lib/wic/plugins/imager/direct.py | 102 +++++++++++++++++++++-- scripts/wic =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0| =C2=A0=C2=A02 +- 4 files changed, 98 insertions(+), 9 deletions(-) diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py index 0df9eb0d05..45db2eaa5d 100644 --- a/scripts/lib/wic/ksparser.py +++ b/scripts/lib/wic/ksparser.py @@ -148,6 +148,7 @@ class KickStart(): part =3D subparsers.add_parser('part') part.add_argument('mountpoint', nargs=3D'?') part.add_argument('--active', action=3D'store_true') + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0part.add_argument('--fixed-alig= n', action=3D'store_true') part.add_argument('--align', type=3Dint) part.add_argument('--offset', type=3Dsizetype("K", True)) part.add_argument('--exclude-path', nargs=3D'+') @@ -194,6 +195,7 @@ class KickStart(): default=3D'msdos') bootloader.add_argument('--timeout', type=3Dint) bootloader.add_argument('--source') + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bootloader.add_argument('--main= -ptable-offset', type=3Dsizetype("K"), default=3D0) include =3D subparsers.add_parser('include') include.add_argument('path', type=3Dcannedpathtype) diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index 5275da6ed3..1bd2b457cd 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -26,6 +26,7 @@ class Partition(): self.args =3D args self.active =3D args.active self.align =3D args.align + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.fixed_align =3D args.fixed= _align self.disk =3D args.disk self.device =3D None self.extra_space =3D args.extra_space diff --git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plu= gins/imager/direct.py index 165fc2979f..efa4d7804e 100644 --- a/scripts/lib/wic/plugins/imager/direct.py +++ b/scripts/lib/wic/plugins/imager/direct.py @@ -16,6 +16,7 @@ import random import shutil import tempfile import uuid +import operator from time import strftime @@ -66,6 +67,7 @@ class DirectPlugin(ImagerPlugin): self.workdir =3D self.setup_workdir(options.workdir) self._image =3D None self.ptable_format =3D self.ks.bootloader.ptable + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.main_ptable_offset =3D sel= f.ks.bootloader.main_ptable_offset self.parts =3D self.ks.partitions # as a convenience, set source to the boot partition source @@ -77,6 +79,7 @@ class DirectPlugin(ImagerPlugin): image_path =3D self._full_path(self.workdir, self.parts[0].disk, "direct") self._image =3D PartitionedImage(image_path, self.ptable_format, + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0self.main_ptable_offset, self.parts, self.native_sysroot, options.extra_space) @@ -294,15 +297,27 @@ GPT_OVERHEAD =3D 34 # Size of a sector in bytes SECTOR_SIZE =3D 512 +class memoryRegion(): + =C2=A0=C2=A0=C2=A0def __init__(self): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.start_sector =3D 1 + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.end_sector =3D 1 + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.part_num =3D 0 + + =C2=A0=C2=A0=C2=A0def __init__(self, part_num, start_sec, end_sec): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.part_num =3D part_num + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.start_sector =3D start_sec + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.end_sector =3D end_sec + class PartitionedImage(): """ Partitioned image in a file. """ - =C2=A0=C2=A0=C2=A0def __init__(self, path, ptable_format, partitions, nat= ive_sysroot=3DNone, extra_space=3D0): + =C2=A0=C2=A0=C2=A0def __init__(self, path, ptable_format, main_ptable_off= set, partitions, native_sysroot=3DNone, extra_space=3D0): self.path =3D path =C2=A0# Path to the image file self.numpart =3D 0 =C2=A0# Number of allocated partitions self.realpart =3D 0 # Number of partitions in the partition table + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.fixedalign_part =3D 0 # Nu= mber of partitions with absolute alignment self.primary_part_num =3D 0 =C2=A0# Number of primary partitions (msdos) self.extendedpart =3D 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# Create extended par= tition before this logical partition (msdos) self.extended_size_sec =3D 0 # Size of exteded partition (msdos) @@ -311,11 +326,13 @@ class PartitionedImage(): self.min_size =3D 0 # Minimum required disk size to fit # all partitions (in bytes) self.ptable_format =3D ptable_format =C2=A0# Partition table format + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.main_ptable_offset =3D mai= n_ptable_offset # for GPT table, offset # Disk system identifier self.identifier =3D random.SystemRandom().randint(1, 0xffffffff) self.partitions =3D partitions self.partimages =3D [] + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.used_memory_region =3D [] # Size of a sector used in calculations self.sector_size =3D SECTOR_SIZE self.native_sysroot =3D native_sysroot @@ -368,6 +385,30 @@ class PartitionedImage(): # Converting kB to sectors for parted part.size_sec =3D part.disk_size * 1024 // self.sector_size + =C2=A0=C2=A0=C2=A0def _add_busy_region(self, part_num, start_sec, end_sec= ): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.used_memory_region.append(= memoryRegion(part_num, start_sec, end_sec)) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.used_memory_region =3D sor= ted(self.used_memory_region, key=3Doperator.attrgetter('start_sector')) + + =C2=A0=C2=A0=C2=A0def _print_busy_region_list(self): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0list =3D "Busy memory region (i= n sector)\n" + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0list +=3D "\tpart#\tstart\tEnd\= n" + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0for num in range(len(self.used_= memory_region)): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0list += =3D"\t%s\t%s\t%s\n" % (self.used_memory_region[num].part_num, \ + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0self.used_memory_region[num].start_sector, self.used_memo= ry_region[num].end_sector) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0logger.debug(list) + + =C2=A0=C2=A0=C2=A0def _check_memory_region(self, start_sec, end_sec ): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0""" Check if the specified regi= on is already flagges as used. Returns + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0zero if not used, 1 otherwise""= " + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0for num in range(len(self.used_= memory_region)): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0region = =3D self.used_memory_region[num] + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if ( ( = region.start_sector < start_sec and start_sec < region.end_sector) or \ + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0( region.start_sector < end_sec a= nd end_sec < region.end_sector) ): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0return 1 + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if ( st= art_sec < region.start_sector and end_sec > region.end_sector ): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0return 1 + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0return 0 + def layout_partitions(self): """ Layout the partitions, meaning calculate the position of every partition on the disk. The 'ptable_format' parameter defines the @@ -379,6 +420,35 @@ class PartitionedImage(): # partitions not listed in the table are not included. num_real_partitions =3D len([p for p in self.partitions if not p.no_table]) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# The partitions flagged as fix= ed alignment partition are placed into image + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# before the others. Instead, r= elative-aligned partitions are placed in a + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# serialized fashion, with cont= rol of the intersection of memory areas. + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# The base address of the relat= ive-aligned partitions is, as before, + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# the maximum address accupied = by the partition table + + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# Flag as used the momoery requ= ired for partition table + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self._add_busy_region(-1, 0, MB= R_OVERHEAD) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if self.ptable_format =3D=3D "g= pt": + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0offset = =3D self.main_ptable_offset * 1024 // self.sector_size + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self._a= dd_busy_region(-1, offset, offset + GPT_OVERHEAD) + + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# Search all partition with fix= ed position into the image + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0for num in range(len(self.parti= tions)): + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0part = =3D self.partitions[num] + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if not = part.fixed_align: + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0continue + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if not = part.no_table: + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0raise WicError("A partition with fixed alignment must hav= e no_table flag set") + + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0start_s= ector =3D (part.align * 1024 // self.sector_size) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0end_sec= tor =3D start_sector + part.size_sec - 1 + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if self= ._check_memory_region(start_sector, end_sector) =3D=3D 1: + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0self._print_busy_region_list() + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0raise WicError("A partition wants to use an already used = memory region (sectors %d - %d)" \ + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0% (start_= sector, end_sector)) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self._a= dd_busy_region(num, start_sector, end_sector) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0part.st= art =3D start_sector + # Go through partitions in the order they are added in .ks file for num in range(len(self.partitions)): part =3D self.partitions[num] @@ -403,7 +473,7 @@ class PartitionedImage(): if self.ptable_format =3D=3D "msdos": overhead =3D MBR_OVERHEAD elif self.ptable_format =3D=3D "gpt": - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0overhead =3D GPT_OVERHEAD + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0overhead =3D (self.main_ptable_of= fset * 1024 // self.sector_size) + GPT_OVERHEAD # Skip one sector required for the partitioning scheme overhead self.offset +=3D overhead @@ -418,7 +488,7 @@ class PartitionedImage(): self.offset +=3D 2 align_sectors =3D 0 - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if part= .align: + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if not = part.fixed_align and part.align: # If not first partition and we do have alignment set we need # to align the partition. # FIXME: This leaves a empty spaces to the disk. To fill the @@ -440,7 +510,7 @@ class PartitionedImage(): # increase the offset so we actually start the partition on right alignment self.offset +=3D align_sectors - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if part= .offset is not None: + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if not = part.fixed_align and part.offset is not None: offset =3D part.offset // self.sector_size if offset * self.sector_size !=3D part.offset: @@ -455,14 +525,20 @@ class PartitionedImage(): self.offset =3D offset - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0part.st= art =3D self.offset - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.of= fset +=3D part.size_sec - if not part.no_table: part.num =3D self.realpart else: part.num =3D 0 + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if not = part.fixed_align: + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0part.start =3D self.offset + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0self.offset +=3D part.size_sec + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0if self._check_memory_region(part.start, self.offset) =3D= =3D 1: + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self._print_busy_region_list() + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0raise WicError("A partition wants= to use an already used memory region (sectors %d - %d)" \ + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0% (part.s= tart, self.offset)) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0self._add_busy_region(part.num, part.start, self.offset) + if self.ptable_format =3D=3D "msdos" and not part.no_table: if part.type =3D=3D 'logical': self.logical_part_cnt +=3D 1 @@ -482,12 +558,14 @@ class PartitionedImage(): "sectors (%d bytes).", part.mountpoint, part.disk, part.num, part.start, self.offset - 1, part.size_sec, part.size_sec * self.sector_size) + + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self._print_busy_region_list() # Once all the partitions have been layed out, we can calculate the # minumim disk size self.min_size =3D self.offset if self.ptable_format =3D=3D "gpt": - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.mi= n_size +=3D GPT_OVERHEAD + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0self.mi= n_size +=3D (self.main_ptable_offset * 1024 // self.sector_size) + GPT_OVER= HEAD self.min_size *=3D self.sector_size self.min_size +=3D self.extra_space @@ -606,6 +684,14 @@ class PartitionedImage(): exec_native_cmd("sfdisk --part-type %s %s %s" % \ (self.path, part.num, part.system_id), self.native_sysroot) + + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if self.ptable_format =3D=3D "g= pt" and self.main_ptable_offset > 0: + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0main_pt= able_sectors =3D self.main_ptable_offset * 1024 // self.sector_size + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0logger.= debug("Move the main GPT partition table forward by %s sector(s)", main_pta= ble_sectors) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0cmd =3D= "sgdisk -j %d %s" % (main_ptable_sectors, self.path) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0#cmd = =3D "(echo -e 'x'; echo -e 'j' ; echo -e '%d'; echo -e 'w'; echo -e 'Y') | = gdisk %s" % (main_ptable_sectors, self.path) + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0exec_na= tive_cmd(cmd, self.native_sysroot) + def cleanup(self): pass diff --git a/scripts/wic b/scripts/wic index 06e0b48db0..0c59a550ef 100755 --- a/scripts/wic +++ b/scripts/wic @@ -59,7 +59,7 @@ from wic import help as hlp def wic_logger(): - =C2=A0=C2=A0=C2=A0"""Create and convfigure wic logger.""" + =C2=A0=C2=A0=C2=A0"""Create and configure wic logger.""" logger =3D logging.getLogger('wic') logger.setLevel(logging.INFO) -- 2.39.2 --TrCFxH6zZF2drAJkL235 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Iss= ue:
the usage of a gpt table with i.MX processor is not compat= ible by default.
This is due to a memory conflict: the main partition= table is placed into
the same memory space used to flash the header = of the U-Boot. This last
can not be moved since the ROM code of the i= .MX6 starts to read to that
fixed address (0x400).
This problem= can be solved using a feature of the GTP partition table for
witch t= he main table can be moved to a desired memory area.
The WIC library = present in Poky manages the GTP table through some
standard Linux com= mands (gdisk and sgdisk) which ones have already the
opportunity to p= erform this task.
This patch introduces this feature with the additio= n of some wic's options,
used also in wks file.

New parti= tion placement behavior:
Since now all partitions are placed after th= e gpt table presents into the
first 1KB of memory (including both the= gpt header at the first byte of
memory and the main partition table = starting at 0x400).
Now the main gpt table can be placed in other add= ress spaces and a partition
can be placed also before the gpt table. = Recap:
- by default, all partitions are placed sequentially, one afte= r the other.
The first partition implicitly starts after the GPT part= ition table.
- adding a flag to the item of a partition into the wks = file, the partition
can be placed, starting from a fixed memory offse= t (neglecting the memory
space occupied by the GTP table).
Comb= ining the moving of the main partition table and the absolute (not yet
relative) positioning of a partition, now it is possible to place one or = more
   partitions before the GPT table.

N= ew flags:
- main-ptable-offset (size in KB): to add as property of th= e bootloader item.
Specifies the main partition table location in KB.=
- fixed-align (boolean): to add as property of a partition. Specifie= s that the
partition must to placed with absolute offset, used in con= junction with --align
property

Example:
setting use= d for a wic for an i.MX6 processor, where the main partition table is
placed with the absolute offset of 4MB

bootloader --ptable gpt= --main-ptable-offset 4096K
part / --source rawcopy --sourceparams=3D= "file=3Du-boot.imx" --ondisk mmcblk --align 1 --size 1M --no-table --fixed-= align

Test environment:
This patch is a part of Edgehog O= S (https://git.seco.com/edgehog) a
custom Yocto distribution by SECO = SpA (www.seco.com), used on multiple
architectures (both ARM and x86)= .
This solution is widely used by all SECO customers who use hardware=
solution based on NXP i.MX6 processors.

Signed-off-by: D= avide Cardillo <davide.cardillo@seco.com>
Reviewed-by: Tobias P= oganiuch <tobias.poganiuch@seco.com>
Tested-by: Andrea Da Col &= lt;andrea.dacol@seco.com>
Tested-by: Daniel Rinaldi <daniel.rin= aldi@seco.com>
---
scripts/lib/wic/ksparser.py   &= nbsp;          |  &n= bsp;2 +
scripts/lib/wic/partition.py      &n= bsp;      |   1 +
scripts/lib= /wic/plugins/imager/direct.py | 102 +++++++++++++++++++++--
scripts/w= ic             =             &nb= sp;    |   2 +-
4 files changed, 98 ins= ertions(+), 9 deletions(-)

diff --git a/scripts/lib/wic/ksparse= r.py b/scripts/lib/wic/ksparser.py
index 0df9eb0d05..45db2eaa5d 10064= 4
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/kspar= ser.py
@@ -148,6 +148,7 @@ class KickStart():
   = ;     part =3D subparsers.add_parser('part')
        part.add_argument('mountpo= int', nargs=3D'?')
        pa= rt.add_argument('--active', action=3D'store_true')
+   &nbs= p;    part.add_argument('--fixed-align', action=3D'stor= e_true')
        part.add_arg= ument('--align', type=3Dint)
      &nbs= p; part.add_argument('--offset', type=3Dsizetype("K", True))
&nb= sp;       part.add_argument('--exclude-p= ath', nargs=3D'+')
@@ -194,6 +195,7 @@ class KickStart():
 = ;            &n= bsp;            = ;      default=3D'msdos')
  &= nbsp;     bootloader.add_argument('--timeout', typ= e=3Dint)
        bootloader.a= dd_argument('--source')
+        b= ootloader.add_argument('--main-ptable-offset', type=3Dsizetype("K"), defaul= t=3D0)
 
        in= clude =3D subparsers.add_parser('include')
    &n= bsp;   include.add_argument('path', type=3Dcannedpathtype) <= br />diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.= py
index 5275da6ed3..1bd2b457cd 100644
--- a/scripts/lib/wic/pa= rtition.py
+++ b/scripts/lib/wic/partition.py
@@ -26,6 +26,7 @@= class Partition():
        s= elf.args =3D args
        sel= f.active =3D args.active
       &n= bsp;self.align =3D args.align
+       &= nbsp;self.fixed_align =3D args.fixed_align
    &n= bsp;   self.disk =3D args.disk
    = ;    self.device =3D None
    = ;    self.extra_space =3D args.extra_space
diff -= -git a/scripts/lib/wic/plugins/imager/direct.py b/scripts/lib/wic/plugins/i= mager/direct.py
index 165fc2979f..efa4d7804e 100644
--- a/scrip= ts/lib/wic/plugins/imager/direct.py
+++ b/scripts/lib/wic/plugins/ima= ger/direct.py
@@ -16,6 +16,7 @@ import random
import shutil import tempfile
import uuid
+import operator
 from time import strftime
 
@@ -66,6 +67,7 @@ class Dire= ctPlugin(ImagerPlugin):
       &nb= sp;self.workdir =3D self.setup_workdir(options.workdir)
  &= nbsp;     self._image =3D None
  &= nbsp;     self.ptable_format =3D self.ks.bootloade= r.ptable
+        self.main_ptable= _offset =3D self.ks.bootloader.main_ptable_offset
   &= nbsp;    self.parts =3D self.ks.partitions
 =
        # as a convenience, s= et source to the boot partition source
@@ -77,6 +79,7 @@ class Direct= Plugin(ImagerPlugin):
 
      = ;  image_path =3D self._full_path(self.workdir, self.parts[0].dis= k, "direct")
        self._im= age =3D PartitionedImage(image_path, self.ptable_format,
+  &nbs= p;            &= nbsp;           &nbs= p;           self.ma= in_ptable_offset,
        &nb= sp;            =             &nb= sp;     self.parts, self.native_sysroot,
&nb= sp;            =             &nb= sp;            =  options.extra_space)
 
@@ -294,15 +297,27 @@ GPT_OVER= HEAD =3D 34
# Size of a sector in bytes
SECTOR_SIZE =3D 512  
+class memoryRegion():
+    def __init_= _(self):
+        self.start_secto= r =3D 1
+        self.end_sector = =3D 1
+        self.part_num =3D 0=
+
+    def __init__(self, part_num, start_sec, = end_sec):
+        self.part_num = =3D part_num
+        self.start_s= ector =3D start_sec
+        self.= end_sector =3D end_sec
+
class PartitionedImage():
 =    """
    Partitioned image in a = file.
    """
 
-   &nb= sp;def __init__(self, path, ptable_format, partitions, native_sysroot=3DNon= e, extra_space=3D0):
+    def __init__(self, path, pta= ble_format, main_ptable_offset, partitions, native_sysroot=3DNone, extra_sp= ace=3D0):
        self.path = =3D path  # Path to the image file
     = ;   self.numpart =3D 0  # Number of allocated partition= s
        self.realpart =3D 0= # Number of partitions in the partition table
+    &n= bsp;   self.fixedalign_part =3D 0 # Number of partitions wit= h absolute alignment
        = self.primary_part_num =3D 0  # Number of primary partitions (msdos)         self.extendedpart =3D 0=      # Create extended partition before this logi= cal partition (msdos)
        = ;self.extended_size_sec =3D 0 # Size of exteded partition (msdos)
@@ = -311,11 +326,13 @@ class PartitionedImage():
    =     self.min_size =3D 0 # Minimum required disk size to= fit
          &nbs= p;            &= nbsp;  # all partitions (in bytes)
    =     self.ptable_format =3D ptable_format  # Partit= ion table format
+        self.mai= n_ptable_offset =3D main_ptable_offset # for GPT table, offset
 =        # Disk system identifier
&n= bsp;       self.identifier =3D random.Sy= stemRandom().randint(1, 0xffffffff)
 
   &nb= sp;    self.partitions =3D partitions
  = ;      self.partimages =3D []
+  &= nbsp;     self.used_memory_region =3D []
&nb= sp;       # Size of a sector used in cal= culations
        self.sector= _size =3D SECTOR_SIZE
        = ;self.native_sysroot =3D native_sysroot
@@ -368,6 +385,30 @@ class Pa= rtitionedImage():
        &nb= sp;   # Converting kB to sectors for parted
 &nbs= p;          part.size_sec= =3D part.disk_size * 1024 // self.sector_size
 
+  &n= bsp; def _add_busy_region(self, part_num, start_sec, end_sec):
+=        self.used_memory_region.append(m= emoryRegion(part_num, start_sec, end_sec))
+     =    self.used_memory_region =3D sorted(self.used_memory_regio= n, key=3Doperator.attrgetter('start_sector'))
+
+   &= nbsp;def _print_busy_region_list(self):
+     &nb= sp;  list =3D "Busy memory region (in sector)\n"
+  &n= bsp;     list +=3D "\tpart#\tstart\tEnd\n"
+=        for num in range(len(self.used_m= emory_region)):
+         &nb= sp;  list +=3D"\t%s\t%s\t%s\n" % (self.used_memory_region[num].pa= rt_num, \
+          &nb= sp;     self.used_memory_region[num].start_sector,= self.used_memory_region[num].end_sector)
+     &= nbsp;  logger.debug(list)
+
+    def _= check_memory_region(self, start_sec, end_sec ):
+    &= nbsp;   """ Check if the specified region is already flagges= as used. Returns
+        zero if= not used, 1 otherwise"""
+        = ;for num in range(len(self.used_memory_region)):
+    =         region =3D self.used_memory= _region[num]
+          =   if ( ( region.start_sector < start_sec and start_sec < re= gion.end_sector) or \
+        &nb= sp;           ( regi= on.start_sector < end_sec and end_sec < region.end_sector) ):
+=             &n= bsp;  return 1
+        =     if ( start_sec < region.start_sector and end_sec= > region.end_sector ):
+       &nbs= p;        return 1
+  &n= bsp;     return 0
+
   =  def layout_partitions(self):
     &nbs= p;  """ Layout the partitions, meaning calculate the position of = every
        partition on th= e disk. The 'ptable_format' parameter defines the
@@ -379,6 +420,35 @= @ class PartitionedImage():
       = ; # partitions not listed in the table are not included.
 &= nbsp;      num_real_partitions =3D len([p for= p in self.partitions if not p.no_table])
 
+   &= nbsp;    # The partitions flagged as fixed alignment pa= rtition are placed into image
+       &= nbsp;# before the others. Instead, relative-aligned partitions are placed i= n a
+        # serialized fashion,= with control of the intersection of memory areas.
+   &nbs= p;    # The base address of the relative-aligned partit= ions is, as before,
+        # the= maximum address accupied by the partition table
+
+  &nbs= p;     # Flag as used the momoery required for par= tition table
+        self._add_bu= sy_region(-1, 0, MBR_OVERHEAD)
+       =  if self.ptable_format =3D=3D "gpt":
+     &= nbsp;      offset =3D self.main_ptable_offset= * 1024 // self.sector_size
+       &nb= sp;    self._add_busy_region(-1, offset, offset + GPT_O= VERHEAD)
+
+        # Search= all partition with fixed position into the image
+    = ;    for num in range(len(self.partitions)):
+ &n= bsp;          part =3D se= lf.partitions[num]
+         =    if not part.fixed_align:
+     =            continue =
+            i= f not part.no_table:
+        &nbs= p;       raise WicError("A partition wit= h fixed alignment must have no_table flag set")
+
+   = ;         start_sector =3D (pa= rt.align * 1024 // self.sector_size)
+      =       end_sector =3D start_sector + part.size= _sec - 1
+          &nbs= p; if self._check_memory_region(start_sector, end_sector) =3D=3D 1: +            &nb= sp;   self._print_busy_region_list()
+   &nb= sp;            = raise WicError("A partition wants to use an already used memory region (sec= tors %d - %d)" \
+         &n= bsp;            = ;  % (start_sector, end_sector))
+     =        self._add_busy_region(num, start_= sector, end_sector)
+         = ;   part.start =3D start_sector
+
  &n= bsp;     # Go through partitions in the order they= are added in .ks file
       &nbs= p;for num in range(len(self.partitions)):
    &nb= sp;       part =3D self.partitions[num] =
@@ -403,7 +473,7 @@ class PartitionedImage():
   = ;            &n= bsp;if self.ptable_format =3D=3D "msdos":
    &nb= sp;            =    overhead =3D MBR_OVERHEAD
    &= nbsp;           elif= self.ptable_format =3D=3D "gpt":
-      &nb= sp;            =  overhead =3D GPT_OVERHEAD
+       = ;            &n= bsp;overhead =3D (self.main_ptable_offset * 1024 // self.sector_size) + GPT= _OVERHEAD
 
        = ;        # Skip one sector required= for the partitioning scheme overhead
     &= nbsp;          self.offse= t +=3D overhead
@@ -418,7 +488,7 @@ class PartitionedImage():
&= nbsp;           &nbs= p;       self.offset +=3D 2
 =
           &nb= sp;align_sectors =3D 0
-        &n= bsp;   if part.align:
+      =       if not part.fixed_align and part.align:=
           &n= bsp;    # If not first partition and we do have alignme= nt set we need
         =        # to align the partition.
&= nbsp;           &nbs= p;   # FIXME: This leaves a empty spaces to the disk. To fil= l the
@@ -440,7 +510,7 @@ class PartitionedImage():
  = ;            &n= bsp;     # increase the offset so we actually star= t the partition on right alignment
     &nbs= p;            &= nbsp; self.offset +=3D align_sectors
 
-   &= nbsp;        if part.offset is not = None:
+           &= nbsp;if not part.fixed_align and part.offset is not None:
  = ;            &n= bsp; offset =3D part.offset // self.sector_size
 
&nbs= p;            &= nbsp;  if offset * self.sector_size !=3D part.offset:
@@ -4= 55,14 +525,20 @@ class PartitionedImage():
 
  &n= bsp;            = ; self.offset =3D offset
 
-     &= nbsp;      part.start =3D self.offset
-=            self.off= set +=3D part.size_sec
-
      &n= bsp;     if not part.no_table:
  &= nbsp;           &nbs= p; part.num =3D self.realpart
     &nbs= p;      else:
    &= nbsp;           part= .num =3D 0
 
+        &n= bsp;   if not part.fixed_align:
+    &n= bsp;           part.= start =3D self.offset
+        &nb= sp;       self.offset +=3D part.size_sec=
+            =     if self._check_memory_region(part.start, self.offse= t) =3D=3D 1:
+          =           self._print_bus= y_region_list()
+         &nb= sp;          raise WicErr= or("A partition wants to use an already used memory region (sectors %d - %d= )" \
+           &n= bsp;            = ;% (part.start, self.offset))
+       &= nbsp;        self._add_busy_region(= part.num, part.start, self.offset)
+
    &n= bsp;       if self.ptable_format =3D=3D = "msdos" and not part.no_table:
      &n= bsp;         if part.type =3D= =3D 'logical':
         =            self.logi= cal_part_cnt +=3D 1
@@ -482,12 +558,14 @@ class PartitionedImage(): <= br />           &nbs= p;            &= nbsp;"sectors (%d bytes).", part.mountpoint, part.disk,
  &= nbsp;           &nbs= p;          part.num, par= t.start, self.offset - 1, part.size_sec,
    &nbs= p;            &= nbsp;       part.size_sec * self.sector_= size)
+           &= nbsp; 
+        self._print_bu= sy_region_list()
 
      &nbs= p; # Once all the partitions have been layed out, we can calculate the=
        # minumim disk size =
        self.min_size =3D sel= f.offset
        if self.ptab= le_format =3D=3D "gpt":
-        &= nbsp;   self.min_size +=3D GPT_OVERHEAD
+   =          self.min_size +=3D (s= elf.main_ptable_offset * 1024 // self.sector_size) + GPT_OVERHEAD
&nb= sp;
        self.min_size *=3D= self.sector_size
        sel= f.min_size +=3D self.extra_space
@@ -606,6 +684,14 @@ class Partition= edImage():
         &nbs= p;      exec_native_cmd("sfdisk --part-type %= s %s %s" % \
         &n= bsp;            = ;          (self.path, pa= rt.num, part.system_id),
       &n= bsp;            = ;            se= lf.native_sysroot)
+         =         
+    &= nbsp;   if self.ptable_format =3D=3D "gpt" and self.main_pta= ble_offset > 0:
+         =    main_ptable_sectors =3D self.main_ptable_offset * 1024 //= self.sector_size
+         &= nbsp;  logger.debug("Move the main GPT partition table forward by= %s sector(s)", main_ptable_sectors)
+      =       cmd =3D "sgdisk -j %d %s" % (main_ptabl= e_sectors, self.path)
+        &nb= sp;   #cmd =3D "(echo -e 'x'; echo -e 'j' ; echo -e '%d'; ec= ho -e 'w'; echo -e 'Y') | gdisk %s" % (main_ptable_sectors, self.path)
+            exec_= native_cmd(cmd, self.native_sysroot)
+
 
  =   def cleanup(self):
      &n= bsp; pass
diff --git a/scripts/wic b/scripts/wic
index 06e= 0b48db0..0c59a550ef 100755
--- a/scripts/wic
+++ b/scripts/wic =
@@ -59,7 +59,7 @@ from wic import help as hlp
 
 = ;
def wic_logger():
-    """Create and convfigure= wic logger."""
+    """Create and configure wic logge= r."""
    logger =3D logging.getLogger('wic')     logger.setLevel(logging.INFO)
 
= --  
2.39.2

 --TrCFxH6zZF2drAJkL235--