[Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images

Buildroot Archive on lore.kernel.org
 help / color / mirror / Atom feed

* [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images
  2013-12-16 22:26 [Buildroot] [RFC v2] Introduce the genimages infrastructure Yann E. MORIN
@ 2013-12-16 22:26 ` Yann E. MORIN
  0 siblings, 0 replies; 11+ messages in thread
From: Yann E. MORIN @ 2013-12-16 22:26 UTC (permalink / raw)
  To: buildroot

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Contrary to the existing fs/ schemes, which each generate only a single
filesystem image for the root filesystem, this new scheme allows the
user to generate more complex images.

The basis behind this is a .ini-like description of the layout of the
final target storage:
  - the list of device(s)
  - per-device, the list of partition(s)
  - per-partition, the content

It is possible to create MBR- or GPT-based partitoining schemes. Adding
new ones should be relatively easy (but would need adequate host tools).

For now, the only content possible for partitions is a filesystem. It
should be pretty easy to add new types (eg. un-formated, or raw blob).

Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding
more will be relatively easy, provided we have the necessary host
packages to deal with those filesystems.

The existing Buildroot filesystem generators are re-used as much as
possible when it makes sense; when it does not (eg. for vfat), a specific
generator is used.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ryan Barnett <rjbarnet@rockwellcollins.com>
---
 docs/manual/appendix.txt              |   1 +
 docs/manual/customize-filesystems.txt |  35 ++++
 docs/manual/customize.txt             |   2 +
 docs/manual/partition-layout.txt      | 313 ++++++++++++++++++++++++++++++++++
 fs/Config.in                          |   1 +
 fs/custom/Config.in                   |  16 ++
 fs/custom/boot/gpt                    | 126 ++++++++++++++
 fs/custom/boot/mbr                    |  57 +++++++
 fs/custom/boot/pre-post               |   8 +
 fs/custom/custom.mk                   |  38 +++++
 fs/custom/fs/ext                      |  22 +++
 fs/custom/fs/pre-post                 |  67 ++++++++
 fs/custom/fs/vfat                     |  17 ++
 fs/custom/functions                   |  47 +++++
 fs/custom/genimages                   | 242 ++++++++++++++++++++++++++
 15 files changed, 992 insertions(+)
 create mode 100644 docs/manual/customize-filesystems.txt
 create mode 100644 docs/manual/partition-layout.txt
 create mode 100644 fs/custom/Config.in
 create mode 100644 fs/custom/boot/gpt
 create mode 100644 fs/custom/boot/mbr
 create mode 100644 fs/custom/boot/pre-post
 create mode 100644 fs/custom/custom.mk
 create mode 100644 fs/custom/fs/ext
 create mode 100644 fs/custom/fs/pre-post
 create mode 100644 fs/custom/fs/vfat
 create mode 100644 fs/custom/functions
 create mode 100755 fs/custom/genimages

diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
index 74ee8fd..53f4205 100644
--- a/docs/manual/appendix.txt
+++ b/docs/manual/appendix.txt
@@ -6,6 +6,7 @@ Appendix
 
 include::makedev-syntax.txt[]
 include::makeusers-syntax.txt[]
+include::partition-layout.txt[]
 
 
 // Automatically generated lists:
diff --git a/docs/manual/customize-filesystems.txt b/docs/manual/customize-filesystems.txt
new file mode 100644
index 0000000..cddee42
--- /dev/null
+++ b/docs/manual/customize-filesystems.txt
@@ -0,0 +1,35 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[filesystem-custom]]
+Customizing the generated filesystem images
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++Buildroot+ knows by default how to generate a few different kind of
+filesystems, such as +squashfs+, +ext2/3/4+, +cramfs+... But those
+filesystems are all generated to contain the complete target directory
+hierarchy in a single filesystem, mounted as the root filesystem +/+.
+That is, even if you select both an +ext2+ and a +squashfs+ filesystems,
+the content of the two generated images will be the exact same, only the
+types of the filesystems will be different.
+
+Most devices require a more complex setup, with different parts of the
+directory structure split across different filesystems, each stored on
+different partitions of one or more storage devices.
+
++Buildroot+ can generate such complex setups, using a +partition table layout
+description+. This is a simple text file, not unlike the +.ini+ style of
+configuration files, that describes how the target directory hierarchy has
+to be split across the target storage devices. It is a bit like a flattened
+tree of the storage layout.
+
+Set the variable +BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE+ to the path of
+the file containing your +partition table layout description+.
+
+See xref:part-layout-desc-syntax[] for the complete documentation of the
++partition table layout description+ syntax.
+
+[underline]*Note:* Although more versatile than the single filesystem image
+mechanism, the +partition table layout description+ might be unable to
+describe very complex setups. For example, it is not capable of handling
++initramfs+ based systems, or NFS-mounted filesystems.
diff --git a/docs/manual/customize.txt b/docs/manual/customize.txt
index 7e46fd8..6d062ea 100644
--- a/docs/manual/customize.txt
+++ b/docs/manual/customize.txt
@@ -14,6 +14,8 @@ include::customize-kernel-config.txt[]
 
 include::customize-toolchain.txt[]
 
+include::customize-filesystems.txt[]
+
 include::customize-store.txt[]
 
 include::customize-packages.txt[]
diff --git a/docs/manual/partition-layout.txt b/docs/manual/partition-layout.txt
new file mode 100644
index 0000000..d992952
--- /dev/null
+++ b/docs/manual/partition-layout.txt
@@ -0,0 +1,313 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[part-layout-desc-syntax]]
+
+Partition table layout description syntax
+-----------------------------------------
+
+The +partition table layout description+ syntax is not unlike the standard
+https://en.wikipedia.org/wiki/.ini[+.ini+] syntax. There are two types of
+entries: +sections+, that may each contain zero or more +properties+.
+
++Sections+ are specified between square brackets +[]+, _eg._: +[name]+.
+
++Properties+ are specified as key-value pairs, _eg._: +key=value+, and
+are documented as:
+
+* +key-name+ (optional or mandatory): description
+** +value1+: description
+** +value2+: description
+** ...
+
+[underline]*Note:* Unlike the standard +.ini+ syntax, the +partition table
+layout description+ _is_ case-sensitive.
+
+The order of +sections+ is irrelevant. However, for readability, we recomend
+the +partition table layout description+ starts with the +global+ section.
+
+The global section
+~~~~~~~~~~~~~~~~~~
+
+The +[global]+ section defines some global settings, and the list of devices.
+
+Properties for the global section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +extract+ (mandatory): the type of base image to extract
+** +tar+: extract the +rootfs.tar+ base image generated by +Buildroot+
+
+* +devices+ (mandatory): the comma-separated list of storage devices to
+  use on the device. Each device is the filename of the device node
+  present in +/dev+
+
+* +keep_parts+ (optional): also copy the individual partition images
+  of all devices to +$(BINARIES_DIR)+. Settings from the device sections
+  or the partition sections take precedence over this one.
+** +yes+: copy the individual partition images
+** +no+ (the default): do not copy individual partition images
+
+The devices and partitions sections
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The devices and partitions sections define, for each device or partition,
+the content of that device or partition.
+
+For each device listed in the +[global]+ section, there must be a
+corresponding section named after that device.
+
+For each partition listed in a device section, there must be a corresponding
+section named after that partition.
+
+Properties for the device section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +type+ (mandatory): the type of content for that device or partition
+** +boot+: the device contains one or more partitions, and
+   _may_ serve as a boot device
+
+* +keep_parts+ (optional): also copy the individual partition images
+  for this device to +$(BINARIES_DIR)+. Settings from the partition
+  sections take precedence over this one.
+** +yes+: copy the individual partition images
+** +no+ (the default): do not copy individual partition images
+
+Properties for the partition section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +type+ (mandatory): the type of content for that device or partition
+** +fs+: the partition contains a filesystem
+
+* +size+: the size of that partition, in bytes
+
+* +keep+ (optional): copy this partition image to +$(BINARIES_DIR)+
+** +yes+: copy this partition image
+** +no+ (the default): do not copy this partition image
+
+Properties for +type=boot+
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +boot_type+ (mandatory): the partitioning scheme to use on this device
+** +mbr+: use an https://en.wikipedia.org/wiki/Master_boot_record[MBR]
+   partitioning scheme
+** +gpt+: use a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
+   partitioning scheme
+
+* +partitions+ (mandatory): the comma-separated list of partition(s) on
+  this device; no two partitions may have the same name, even if they
+  reside on different devices; partitions names shall match this regexp:
+  `^[[:alpha:]][[:alnum:]-_]*$` (_ie._ starts with a letter, followed by
+  zero or more alpha-numeric character or a dash or an underscore)
+
+* +partalign+ (optional): the alignment of partitions, in bytes; defaults
+  to an alignment of one, which means no alignment; depending on the
+  +boot_type+, some restrictions may apply, and are documented for each
+  +boot_type+
+
+Properties for +boot_type=mbr+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +mbr_bootcode+ (optional): the bootcode to use, as a path to the file
+  containing the bootcode image, relative to the +$(BINARIES_DIR)+
+  directory; defaults to no bootcode (eg. filled with zeroes)
+
+* +partalign+: must be a multiple of 512
+
+Properties for +boot_type=gpt+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +partalign+: must be a multiple of 512
+
+********
+Currently, only 512-byte sectors are supported. 4k sectors are not.
+********
+
+Properties for partitions whose containing device is +boot_type=mbr+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +mbr_type+ (mandatory): the partition
+  https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs[type]
+
+Properties for +type=fs+
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +fs_type+ (mandatory): the type of filesystem to generate
+** +ext+: generate an extended filesystem (ext2, ext3, ext4)
+** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
+
+* +fs_label+ (optional): the label to assign to this filesystem, if that
+  filesystem supports a label
+
+* +fs_files_0+, +fs_files_1+, +fs_files_N+ (optional): the list of files
+  to put on the filesystem. These entries must be indexed starting from
+  0, and must be sequential: the first missing entry ends the list
+
+* +fs_root+ (optional): the mountpoint of the filesystem
+
+* +fs_vfstype+ (optional): the type of filesystem to use when calling
+  +mount+, if different from +fs_type+
+
+* +fs_mntopts+ (optional): the mount options; defaults to +defaults+
+
+_Note_: valid use-cases for +fs_root+ and +fs_files_N+:
+
+* if only +fs_root+ is specified (and no +fs_files_N+): the filesystem
+  content is made exclusively from +$(TARGET_DIR)/$(fs_root)+, and the
+  filesystem is mounted at runtime
+
+* if both +fs_root+ and at least one +fs_files_N+ are specified: the
+  filesystem content is made exclusively from the +fs_files_N+ entries,
+  and mounted at runtime. +$(TARGET_DIR)/$(fs_root)+ must be empty
+
+* if +fs_root+ is not specified, but at least one +fs_files_N+ is: the
+  filesystem content is made exclusively from the +fs_files_N+ entries,
+  and the filesystem is not mounted at runtime
+
+* if neither +fs_root+ nor +fs_files_N+ is specified: this is an error
+
+Properties for +fs_type=ext+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +ext_gen+ (mandatory): the generation of extended filesystem to generate
+** +2+, +3+, +4+: for an ext2, ext3 or ext4 filesystem
+
+* +ext_rev+ (mandatory): the revision of the extended filesystem
+** +0+ (ext2 only): generate a revision 0 extended filesystem filesystem
+** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
+   filesystem
+
+Properties for +fs_type=vfat+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +vfat_size+ (optional): the VFAT-size of the filesystem
+** +12+, +16+, +32+: generate a FAT12, FAT16, or FAT32
+
+Generation of the filesystems
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The filesystems are generated in an order such that a filesystem that is
+mounted as a sub-directory of another filesystem is generated first.
+
+A filesystem is filled with the content of the directory corresponding to
+its mountpoint, and then that directory is emptied before continuing to the
+next filesystem. That way, the mountpoints are empty before the filesystem
+that contains them are generated.
+
+Finally, an entry in added in +/etc/fstab+ for each generated filesystem, so
+they are mounted at boot time.
+
+Requirements on host packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on what types of partitioning and filesystems are used by your
++partition table layout description+, you may have to enable some
+host-packages in the +Host utilities+ sub-menu.
+
+Since the content of a +partition table layout description+ is very
+specific to a board and/or a project, there is no way for Buildroot to
+automatically select those host packages, and thus it is your
+responsibility to select the appropriate ones.
+
+For example, for an MBR partitioning, you will have to enable the +host
+genpart+ package. For FAT filesystems, you will have to enable both of
++host dosfstools+ and +host mtools+.
+
+Examples
+~~~~~~~~
+
+.Simplest partition table layout description
+====
+----
+[global]
+extract=tar
+devices=sda
+
+[sda]
+type=boot
+boot_type=mbr
+partitions=root
+partalign=1048576
+
+[root]
+type=fs
+fs_type=ext
+fs_vfstype=ext4
+fs_root=/
+ext_gen=4
+ext_rev=1
+----
+
+The +partition table layout description+ above defines a single device
++sda+. That device contains a single partition, +root+, with an ext4
+filesystem, which is filled with the whole content of the +rootfs.tar+,
+and is mounted on +/+.
+====
+
+.More copmplex table layout description
+====
+----
+[global]
+extract=tar
+devices=mmcblk0,sda
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1024*1024))
+
+[sda]
+type=boot
+boot_type=mbr
+partitions=data
+partalign=4096
+
+[boot]
+type=fs
+mbr_type=$((0xC))
+size=$((16*1048576))
+fs_type=vfat
+fs_mntopts=ro
+fs_label=BOOT
+fs_root=/boot
+vfat_size=32
+
+[root]
+type=fs
+mbr_type=$((0x83))
+size=268435456
+fs_type=ext
+fs_vfstype=ext4
+fs_mntopts=discard,delalloc
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[data]
+type=fs
+mbr_type=$((0x83))
+size=$((4*1024*1048576))
+fs_type=ext
+fs_vfstype=ext2
+fs_root=/data
+fs_label=DATA
+ext_gen=2
+ext_rev=1
+----
+====
+
+The example above defines two devices, +mmcblk0+ and +sda+.
+
+The +mmcblk0+ device contains two partitions, +boot+ and +root+; partitions
+are aligned on a 1MiB boundary. The +sda+ device contains a single partition,
++data+, aligned on a 4KiB boundary.
+
+The +boot+ partition is a 16MiB FAT32 filesystem filled with the content
+of, and mounted on, +/boot+, and with label +BOOT+.
+
+The +data+ partition is a 4GiB ext2r1 filesystem filled with the content
+of, and mounted on, +/data+, and with label +DATA+.
+
+The +root+ partition is a 256MiB ext4 filesystem filled the the rest of,
+and mounted on, +/+, and with label +ROOT+.
diff --git a/fs/Config.in b/fs/Config.in
index da4c5ff..44e04f7 100644
--- a/fs/Config.in
+++ b/fs/Config.in
@@ -3,6 +3,7 @@ menu "Filesystem images"
 source "fs/cloop/Config.in"
 source "fs/cpio/Config.in"
 source "fs/cramfs/Config.in"
+source "fs/custom/Config.in"
 source "fs/ext2/Config.in"
 source "fs/initramfs/Config.in"
 source "fs/iso9660/Config.in"
diff --git a/fs/custom/Config.in b/fs/custom/Config.in
new file mode 100644
index 0000000..e5a8ee7
--- /dev/null
+++ b/fs/custom/Config.in
@@ -0,0 +1,16 @@
+config BR2_TARGET_ROOTFS_CUSTOM
+	bool "Custom partition table layout"
+	select BR2_TARGET_ROOTFS_TAR
+
+config BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE
+	string "path to the custom partition table layout description"
+	depends on BR2_TARGET_ROOTFS_CUSTOM
+	help
+	  Enter the path to a partition-table for your device.
+
+	  This will allow Buildroot to generate a more complex target
+	  image, which may consist of more than one filesystem on more
+	  than one partition.
+
+	  See docs/manual/bla-bla on how to construct such a partition
+	  table.
diff --git a/fs/custom/boot/gpt b/fs/custom/boot/gpt
new file mode 100644
index 0000000..8592cbf
--- /dev/null
+++ b/fs/custom/boot/gpt
@@ -0,0 +1,126 @@
+# Build a complete GPT-based image
+
+# For a GPT-based partitionning, we need to compute the complete
+# image size before we can attempt to generate the partition table.
+# Then, we need to add the size of the GPT itself, plus that of its
+# backup copy, plus the protective MBR.
+# The size of the GPT itself depends on the sector size, and the
+# number of partitions in the GPT. Sectors can be either 512-byte
+# or 4096-byte large; The numbers of partitions is unlimited, but
+# it is suggested there is space to store at least 128 of them; a
+# partition description is 128-byte large.
+#
+# https://en.wikipedia.org/wiki/GUID_Partition_Table
+#
+# So, here's what we do:
+#   - consider 512-byte sectors (since GPT on 4k sectors is not well
+#     documented)
+#   - consider at least 128 partitions; if the layout defines more than
+#     that, we need to round that number up to the smallest multiple of
+#     4 (since there are 4 partition descriptions in a 512-byte sector)
+#   - generate an empty, sparse file that is big enough to store the MBR,
+#     the two GPT copies, and the aligned partitions.
+#   - dump each partition in turn in their final location in that file
+#   - generate a parted script that creates the partition table in that
+#     file
+#
+#   Simg = 512 + 2*(Sgpt) + ?( aligned(Spart,512) )
+#   Sgpt = 512 + Nent*128
+#
+# Where:
+#   Simg        : size of the image
+#   Sgpt        : size of one GPT
+#   Spart       : size of each partition
+#   Nent        : number of partition entries
+#   aligned()   : the alignment function
+#
+# Sicne 4k-large sectors are not really explained on Wikipedia, we can
+# add this later on.
+
+do_image() {
+    # ${1} is fs_root, irrelevant here
+    local img="${2}"
+
+    # How many partitions do we have?
+    nb_parts=0
+    for part in ${partitions[${dev}]//,/ }; do
+        nb_parts=$((nb_parts+1))
+    done
+
+    # How many partition entries do we need?
+    nb_entries=$((4*((nb_parts+3)/4)))
+    nb_entries=$((nb_entries<128?128:nb_entries))
+
+    # The size of a single GPT
+    gpt_size=$((512+(128*nb_entries)))
+
+    # Offset of the first partition
+    begin=$(align_val $((512+$(align_val ${gpt_size} 512))) ${partalign} )
+
+    # Initialise our image file
+    dd if=/dev/zero of="${img}"     \
+       bs=1 seek=${begin} count=0   \
+       conv=sparse                  2>/dev/null
+
+    # Compute the space required to store all partitions
+    # and store them in the image file
+    size_parts=0
+    _offset=${begin}
+    i=1
+    debug "adding partions descriptions\n"
+    for part in ${partitions[${dev}]//,/ }; do
+        debug "  %s\n" "${part}"
+        _part_img="${tmp_dir}/${dev}.${part}.img"
+        _size=$( align_val $( stat -c '%s' "${_part_img}" ) 512 )
+        part_offset+=( ${_offset} )
+        _attr="${values["${part}:gpt_attr"]}"
+        _label="${values["${part}:gpt_label"]}"
+
+        # If the partition has no label, use the filesystem label
+        if [ -z "${_label}" ]; then
+            _label="${values["${part}:fs_label"]}"
+        fi
+        if [ -z "${_label}" ]; then
+            _label="data"
+        fi
+
+        debug "    start=%s\n" "${_offset}"
+        debug "    size =%s\n" "${_size}"
+        debug "    end  =%s\n" "$((_offset+_size-1))"
+
+        dd if="${_part_img}" of="${img}"    \
+           bs=512 seek=$((_offset/512))     \
+           conv=notrunc,sparse              2>/dev/null
+
+        parted_script+=( mkpart "${_label}"    \
+                                ${_offset}              \
+                                $((_offset+_size-1))    \
+                       )
+        if [ -n "${_attr}" ]; then
+            for attr in "${_attr//,/ }"; do
+                parted_script+=( set ${i} ${attr} on )
+            done
+        fi
+
+        size_parts=$((size_parts+_size))
+        _offset=$((_offset+_size))
+        i=$((i+1))
+    done
+
+    # Terminate our image file
+    img_size=$(align_val $(( begin + size_parts + gpt_size )) 512)
+    debug "begin   =%s\n" ${begin}
+    debug "nb_entry=%s\n" ${nb_entries}
+    debug "gpt_size=%s\n" ${gpt_size}
+    debug "img_size=%s\n" ${img_size}
+    dd if=/dev/zero of="${img}" \
+       bs=1 seek=${img_size}    \
+       count=0 conv=sparse      2>/dev/null
+
+    for i in parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"; do
+        debug "--> '%s'\n" "${i}"
+    done
+    parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"
+}
+
+# vim: ft=sh
diff --git a/fs/custom/boot/mbr b/fs/custom/boot/mbr
new file mode 100644
index 0000000..af7c6cb
--- /dev/null
+++ b/fs/custom/boot/mbr
@@ -0,0 +1,57 @@
+# Build a complete MBR-based image
+
+do_image() {
+    # ${1} is fs_root, irrelevant here
+    local img="${2}"
+    local i begin part part_img size type _begin _size
+    local -a part_offset part_file
+
+    # Fill-in the boot record with zeroes
+    # Ideally, we should dump the bootloader code here, but since
+    # we don't have any so far, that will have to be done in a
+    # later step.
+    debug "adding (fake) bootcode\n"
+    dd if=/dev/zero of="${img}" bs=$((0x1be)) count=0 seek=1 2>/dev/null
+
+    # Generate partition entries
+    i=0
+    begin=${partalign}
+    debug "adding partitions descriptors\n"
+    for part in ${partitions[${dev}]//,/ }; do
+        debug "  %s\n" "${part}"
+        part_offset+=( ${begin} )
+        part_img="${tmp_dir}/${dev}.${part}.img"
+        part_file+=( "${part_img}" )
+        size=$( align_val $( stat -c '%s' "${part_img}" ) 512 )
+        type="${values["${part}:mbr_type"]}"
+        # LBA is exressed in a number of 512-byte blocks
+        # and genparts only deals with LBA
+        _begin=$((begin/512))   # begin is already 512-byte aligned
+        _size=$((size/512))     # size is already 512-byte aligned
+        debug "    start=%s (LBA %s)\n" "${begin}" "${_begin}"
+        debug "    size =%s (LBA %s)\n" "${size}"  "${_size}"
+        debug "    type =%s\n"          "${type}"
+        genpart -b ${_begin} -s ${_size} -t ${type} >>"${img}"
+        begin=$( align_val $((begin+size)) ${partalign} )
+        i=$((i+1))
+    done
+    nb_parts=${i}
+    # Generate entries for empty partitions
+    for(( ; i<4; i++ )); do
+        debug "  (empty)\n"
+        genpart -t 0 >>"${img}"
+    done
+    # Dump the boot signature
+    printf "\x55\xaa" >>"${img}"
+
+    for(( i=0; i<nb_parts; i++ )); do
+        part_img="${part_file[${i}]}"
+        offset=${part_offset[${i}]}
+        _offset=$(( offset/512 ))  # offset is already 512-byte aligned
+        dd if="${part_img}" of="${img}" \
+           bs=512 seek=${_offset}       \
+           conv=notrunc,sparse          2>/dev/null
+    done
+}
+
+# vim: ft=sh
diff --git a/fs/custom/boot/pre-post b/fs/custom/boot/pre-post
new file mode 100644
index 0000000..507d17e
--- /dev/null
+++ b/fs/custom/boot/pre-post
@@ -0,0 +1,8 @@
+do_image_pre() {
+    :
+}
+do_image_post() {
+    :
+}
+
+#vim: set ft=sh
diff --git a/fs/custom/custom.mk b/fs/custom/custom.mk
new file mode 100644
index 0000000..63c1f23
--- /dev/null
+++ b/fs/custom/custom.mk
@@ -0,0 +1,38 @@
+################################################################################
+#
+# custom partitioning
+#
+################################################################################
+
+define ROOTFS_CUSTOM_CMD
+	BUILD_DIR=$(BUILD_DIR) fs/custom/genimages \
+		'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
+endef
+
+# rootfs-custom uses rootfs.tar as the source to generate
+# the resulting image(s), so we need to build it first.
+ROOTFS_CUSTOM_DEPENDENCIES += rootfs-tar
+
+# All of the following filesystem generators, or partition managers, are
+# optional, but if they are selected, we may need them, so we need to
+# depend on them
+ifeq ($(BR2_PACKAGE_HOST_DOSFSTOOLS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-dosfstools
+endif
+ifeq ($(BR2_PACKAGE_HOST_E2FSPROGS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-e2fsprogs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENEXT2FS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-genext2fs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENPART),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-genpart
+endif
+ifeq ($(BR2_PACKAGE_HOST_MTOOLS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-mtools
+endif
+ifeq ($(BR2_PACKAGE_HOST_PARTED),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-parted
+endif
+
+$(eval $(call ROOTFS_TARGET,custom))
diff --git a/fs/custom/fs/ext b/fs/custom/fs/ext
new file mode 100644
index 0000000..5f9b4e7
--- /dev/null
+++ b/fs/custom/fs/ext
@@ -0,0 +1,22 @@
+# Create an extended file system
+
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+    local gen rev
+
+    fs_opts+=( -z )
+    fs_opts+=( -d "${root_dir}" )
+    [ -z "${size}"    ] || fs_opts+=( -b $((size/1024)) )
+    [ -n "${ext_gen}" ] || ext_gen=2
+    [ -n "${ext_rev}" ] || ext_rev=1
+
+    # Remember, we're running from Buildroot's TOP_DIR
+    GEN=${ext_gen} REV=${ext_rev}                   \
+    ./fs/ext2/genext2fs.sh "${fs_opts[@]}" "${img}" >/dev/null
+
+    [ -z "${fs_label}" ] || tune2fs -L "${fs_label}" "${img}" >/dev/null
+}
+
+# vim: ft=sh
diff --git a/fs/custom/fs/pre-post b/fs/custom/fs/pre-post
new file mode 100644
index 0000000..40ec047
--- /dev/null
+++ b/fs/custom/fs/pre-post
@@ -0,0 +1,67 @@
+#-----------------------------------------------------------------------------
+do_image_pre() {
+    local i file
+
+    # if fs_root_dir is not specified, we have to create one
+    # It *does* override the caller's fs_root_dir value, but
+    # that's on purpose
+    # If fs_root_dir is specified, and we have at least fs_files_0,
+    # then fs_root_dir/ must be enpty
+    if [ -z "${fs_root_dir}" ]; then
+        if [ -n "${values["${part}:fs_files_0"]}" ]; then
+            error "%s: no fs_root specified, and no fs_files_0\n" "${part}"
+        fi
+        fs_root_dir="$( mktemp -d "${tmp_dir}/XXXXXX" )"
+    else
+        if [    -n "${values["${part}:fs_files_0"]}"      \
+             -a $( ls -1A "${fs_root_dir}" |wc -l ) -ne 0 ]; then
+            error "%s: %s is not empty, but fs_files_0 is specified\n"  \
+                  "${part}" "${fs_root_dir#${fs_root}}"
+        fi
+    fi
+
+    i=0
+    while true; do
+        file="${values["${part}:fs_files_${i}"]}"
+        [ -n "${file}" ] || break
+        debug "%s: adding fs_files_%d %s\n" "${part}" ${i} "${file}"
+        install -D "${BINARIES_DIR}/${file}" "${fs_root_dir}/${file##*/}"
+        i=$((i+1))
+    done
+}
+
+#-----------------------------------------------------------------------------
+do_image_post() {
+    local rootfs_dir="${1}"
+    local fs_root="${2}"
+    local img_file="${3}"
+    local part="${4}"
+    local dev mntops vfstype fs_root_esc
+
+    subname+="[post-image]"
+
+    # Empty the partition's mountpoint
+    find "${fs_root_dir}" -maxdepth 1 \! -path "${fs_root_dir}" -exec rm -rf {} +
+
+    # Add entry in fstab, but not if this is '/'
+    # Don't add either if rootfs was not extracted
+    if [    "${fs_root}" = "/" -o -z "${fs_root}" \
+         -o -z "${values["global:extract"]}"      ]; then
+        return 0
+    fi
+    fs_root_esc="$( sed -r -e 's:/:\\/:g;' <<<"${fs_root}" )"
+    sed -r -i -e "/[^[:space:]]+[[:space:]]+${fs_root_esc}[[:space:]]/d"    \
+                 "${rootfs_dir}/etc/fstab"
+    dev="$( get_part_dev_node "${part}" )"
+    vfstype="${fs_vfstype:-${fs_type}}"
+    mntops="${fs_mntops:-defaults}"
+    printf "/dev/%s %s %s %s 0 0\n"     \
+           "${dev}" "${fs_root}"        \
+           "${vfstype}" "${mntops}"     \
+           >>"${rootfs_dir}/etc/fstab"
+
+    subname="${subname%\[post-image\]}"
+}
+
+#-----------------------------------------------------------------------------
+# vim: ft=sh
diff --git a/fs/custom/fs/vfat b/fs/custom/fs/vfat
new file mode 100644
index 0000000..aa5545f
--- /dev/null
+++ b/fs/custom/fs/vfat
@@ -0,0 +1,17 @@
+# Create a VFAT file system
+
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+
+    dd if=/dev/zero of="${img}" bs=${size} count=0 seek=1 2>/dev/null
+
+    [ -z "${vfat_size}" ] || fs_opts+=( -F ${vfat_size} )
+    [ -z "${fs_label}"  ] || fs_opts+=( -n "${fs_label}" )
+    mkfs.vfat "${fs_opts[@]}" "${img}" >/dev/null
+
+    mcopy -i "${img}" "${root_dir}/"* '::'
+}
+
+# vim: ft=sh
diff --git a/fs/custom/functions b/fs/custom/functions
new file mode 100644
index 0000000..4b41021
--- /dev/null
+++ b/fs/custom/functions
@@ -0,0 +1,47 @@
+# Common functions
+
+#------------------------------------------------------------------------------
+align_val() {
+    local val="${1}"
+    local align="${2}"
+    local aligned
+
+    aligned=$(( ( (val+align-1) / align ) * align ))
+
+    printf "%d" ${aligned}
+}
+
+#------------------------------------------------------------------------------
+# Some trace functions
+trace() {
+    local fmt="${1}"
+    shift
+
+    printf "%s" "${myname}"
+    if [ -n "${subname}" ]; then
+        printf "(%s)" "${subname}"
+    fi
+    printf ": ${fmt}" "${@}"
+}
+
+debug() { :; }
+if [ -n "${DEBUG}" ]; then
+    debug() {
+        trace "${@}" >&2
+    }
+fi
+
+error() {
+    trace "${@}" >&2
+    exit 1
+}
+
+on_error() {
+    local ret=${?}
+
+    error "unexpected error caught: %d\n" ${ret}
+}
+trap on_error ERR
+set -E -e
+
+# vim: ft=sh
diff --git a/fs/custom/genimages b/fs/custom/genimages
new file mode 100755
index 0000000..00820fc
--- /dev/null
+++ b/fs/custom/genimages
@@ -0,0 +1,242 @@
+#!/bin/bash
+set -e
+set -E
+
+#-----------------------------------------------------------------------------
+main() {
+    local part_table="${1}"
+    local tmp_dir
+    local rootfs_dir
+    local -a devices
+    local extract
+    local cur_section
+    local -a sections devices partitions
+    local -A variables values partdevs
+    local sec dev part var val
+    local secs devs parts vars vals
+    local has_global_section
+
+    # We need bash 4 or above for asociative arrays
+    if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
+        error "bash 4 or above is needed\n"
+    fi
+
+    if [ ! -f "${part_table}" ]; then
+        error "%s: no such file\n" "${part_table}"
+        exit 1
+    fi
+
+    export PATH="${HOST_DIR}/usr/bin:${HOST_DIR}/usr/sbin:${PATH}"
+
+    # Parse all the sections in one go, we'll sort
+    # all the mess afterwards...
+    debug "parsing partitions descriptions file '%s'\n" \
+          "${part_table}"
+    has_global_section=0
+    while read line; do
+        line="$( sed -r -e 's/[[:space:]]*#.*$//;' <<<"${line}" )"
+
+        # Detect start of global section, skip anything else
+        case "${line}" in
+        "") continue ;;
+        '[global]') has_global_section=1 ;;&
+        '['*']')
+            cur_section="$( sed -r -e 's/[][]//g;' <<<"${line}" )"
+            debug "  entering section '%s'\n" "${cur_section}"
+            sections+=( "${cur_section}" )
+            continue
+        ;;
+        ?*=*) ;;
+        *)  error "malformed entry '%s'\n" "${line}" ;;
+        esac
+
+        var="${line%%=*}"
+        eval val="${line#*=}"
+        debug "    adding '%s'='%s'\n" "${var}" "${val}"
+        variables+=( ["${cur_section}"]=",${var}" )
+        values+=( ["${cur_section}:${var}"]="${val}" )
+    done <"${part_table}"
+
+    if [ ${has_global_section} -eq 0 ]; then
+        error "no global section defined\n"
+    fi
+
+    # Create lists of devices, partitions, and partition:device pairs.
+    debug "creating intermediate lists\n"
+    devices=( ${values["global:devices"]//,/ } )
+    for dev in "${devices[@]}"; do
+        # Sanity check first: all devices must have a corresponding
+        # section, which means they should have a type
+        if [ -z "${values["${dev}:type"]}" ]; then
+            error "device '%s' has no type (no section?)\n" "${dev}"
+        fi
+        partitions+=( ${values["${dev}:partitions"]//,/ } )
+        for part in ${values["${dev}:partitions"]//,/ }; do
+            # Sanity check first: all partitions must have a corresponding
+            # section, which means they should have a type
+            if [ -z "${values["${part}:type"]}" ]; then
+                error "partition '%s' has no type (no section?)\n" "${dev}"
+            fi
+            partdevs+=( ["${part}"]="${dev}" )
+        done
+    done
+
+    # Now, we must order the partitions so that their mountpoint
+    # is empty by the time we build the upper-level partition.
+    # For example, given this layout of mountpoints:
+    #   /
+    #   /usr
+    #   /usr/var
+    # We must ensure /usr/var is empty@the time we create the /usr
+    # filesystem image; and similarly, we must ensure /usr is empty by
+    # the time we create the / filesystem image
+    # So, a simple reverse alphabetical sort will do the trick
+    debug "sorting partitions\n"
+    sorted_parts=( $(
+        for part in "${partitions[@]}"; do
+            # Partitions that are not mounted can be generated
+            # in any order
+            if [ -n "${values["${part}:fs_root"]}" ]; then
+                printf "%s:%s\n" "${part}" "${values["${part}:fs_root"]}"
+            else
+                printf "%s\n" "${part}"
+            fi
+        done                    \
+        |sort -t: -k2 -r        \
+        |sed -r -e 's/:[^:]+$//;'
+    ) )
+
+    tmp_dir="${BUILD_DIR}/genimages.tmp"
+    rootfs_dir="${tmp_dir}/rootfs"
+    # Since we don't remove it in case of error (to be able to inspect its
+    # content), we must remove it now (a previous run may have left it).
+    rm -rf "${tmp_dir}"
+    mkdir -p "${rootfs_dir}"
+
+    case "${values["global:extract"]}" in
+        tar)
+            trace "extracting rootfs.tar\n"
+            tar xf "${BINARIES_DIR}/rootfs.tar" -C "${rootfs_dir}"
+        ;;
+        *)  error "unknown extract method '%s'\n" "${extract:-(none)}"
+        ;;
+    esac
+
+    # Render all partition images
+    for part in "${sorted_parts[@]}"; do
+        trace "preparing filesystem for partition '%s'\n" "${part}"
+        render_img "${rootfs_dir}" "${part}"                        \
+                   "${tmp_dir}/${partdevs["${part}"]}.${part}.img"
+    done
+
+    # Aggregate all devices images
+    for dev in "${devices[@]}"; do
+        trace "assembling partitions in device '%s'\n" "${dev}"
+        render_img "${rootfs_dir}" "${dev}" "${tmp_dir}/${dev}.img"
+    done
+
+    # Copy all partitions and devices images to the image dir
+    if [ "${values["global:keep_partitions"]}" = "yes" ]; then
+        for part in "${sorted_parts[@]}"; do
+            debug "copying partition '%s' to image dir\n" "${part}"
+            dd if="${tmp_dir}/${partdevs["${part}"]}.${part}.img"           \
+               of="${BINARIES_DIR}/$( get_part_dev_node "${part}" ).img"    \
+               bs=4096 conv=sparse                                          2>/dev/null
+        done
+    fi
+    for dev in "${devices[@]}"; do
+        debug "copying device '%s' to image dir\n" "${dev}"
+        dd if="${tmp_dir}/${dev}.img"       \
+           of="${BINARIES_DIR}/${dev}.img"  \
+           bs=4096 conv=sparse              2>/dev/null
+    done
+
+    [ -n "${DEBUG}" ] || rm -rf "${tmp_dir}"
+}
+
+#-----------------------------------------------------------------------------
+render_img() {
+    local rootfs_dir="${1}"
+    local img="${2}"
+    local img_file="${3}"
+    local type sub_type fs_root_dir
+
+    type="${values["${img}:type"]}"
+    sub_type="${values["${img}:${type}_type"]}"
+
+    # Sanity checks
+    [ -n "${type}" ] || error "'%s': unspecified type\n" "${img}"
+    if [ ! -d "fs/custom/${type}" ]; then
+        error "'%s': unsupported type '%s'\n" "${img}" "${type}"
+    fi
+    [ -n "${sub_type}" ] || error "'%s': unspecified %s_type\n" "${img}" "${type}"
+    if [ ! -f "fs/custom/${type}/${sub_type}" ]; then
+        error "'%s': unknown %s_type '%s'\n" "${img}" "${type}" "${sub_type}"
+    fi
+
+    # Need to call the renderer in a subshell so that its definitions
+    # do not pollute our environment
+    subname="${sub_type}"
+    (
+        trap 'exit $?' ERR
+        for var in ${variables["${img}"]//,/ }; do
+            eval "${var}=\"${values["${img}:${var}"]}\""
+        done
+        fs_root_dir="${rootfs_dir}${fs_root}"
+        . "fs/custom/${type}/pre-post"
+        . "fs/custom/${type}/${sub_type}"
+        do_image_pre "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
+        do_image "${fs_root_dir}" "${img_file}"
+        do_image_post "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
+    )
+    ret=${?}
+    [ ${ret} -eq 0 ] || exit ${ret}
+    subname=""
+}
+
+#-----------------------------------------------------------------------------
+get_part_dev_node() {
+    local part="${1}"
+    local dev
+    local i c p
+
+    dev="${partdevs["${part}"]}"
+    i="${values["${dev}:partstart"]:-1}"
+
+    # If device node ends with a number, partitions are denoted
+    # with a 'p' before the partition number, eg.:
+    #   /dev/mmcblk0    --> /dev/mmcblk0p1
+    #   /dev/sda        --> /dev/sda1
+    case "${dev#${dev%?}}" in
+        [0-9])  c="p";;
+        *)      c="";;
+    esac
+
+    for p in ${values["${dev}:partitions"]//,/ }; do
+        if [ "${p}" = "${part}" ]; then
+            printf "%s%s%d" "${dev}" "${c}" ${i}
+            return 0
+        fi
+        i=$((i+1))
+    done
+
+    error "'%s': partition not found. WTF?\n" "${part}"
+}
+
+#-----------------------------------------------------------------------------
+myname="${0##*/}"
+mydir="${0%/*}"
+
+TOP_DIR="$( pwd )"
+export myname mydir TOP_DIR
+
+. "fs/custom/functions"
+
+# This script can deal with extracting the rootfs tarball, but we need to
+# be root for that.
+if [ $(id -u) -ne 0 ]; then
+    printf "error: not root\n"
+    exit 1
+else
+    main "${@}"
+fi
-- 
1.8.1.2

^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [Buildroot] [pull request v3] Pull request for branch yem/genimages
@ 2013-12-20 22:32 Yann E. MORIN
  2013-12-20 22:32 ` [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images Yann E. MORIN
                   ` (3 more replies)
  0 siblings, 4 replies; 11+ messages in thread
From: Yann E. MORIN @ 2013-12-20 22:32 UTC (permalink / raw)
  To: buildroot

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Hello All!

During the last developpers' day in Edimburgh, we discussed my Buildroot.config
project, and especially the 'genimages' infrastructure I've done in there.

I've came up with 'genimages' after I came to the conclusion that the current
images generation in Buildroot is limited to generating a single filesystem
images, but most device do require a more complex setup.

For example. the Rapsberry Pi needs an SDcard (mmcblk0) formatted as thus:

    mmcblk0
    |- mmcblk0p1  (FAT32)
    |  |- bootcode.bin
    |  |- fixup.dat
    |  |- start.elf
    |  |- config.txt
    |  |- cmdline.txt
    |  `- kernel.bin
    `- mmcblk0p2  (whatever, eg. ext3, btrfs...)
       `- all of /

With the current output of Buildroot, that requires a lot of fiddling with
the partitioning of the SDcard (may require root), formatting the filesystems
(may require root), extract rootfs.tar (may require root to mount, requires
root to extract), copy select files from output/images/rpi-firmware/ to the
first partition (may require root to mount).

Each user would either have to manually perform those actions (error prone),
or write a small shell script to automate this (not trivial).

Also, some other devices require special, but different formatting as well.

Rather than have every users all suffer in silence (and sometimes not in
silence) by re-inventing such a script over and over again, for each device
that need such a setup, I came up with the 'genimages' idea: a description
of the layout of the storage devices, partitioning schemes, and partitions
content.

So, here is the 'genimages' from Buildroot.config, that I updated (in fact,
trimmed) to include in Buildroot itself.

The basis is that we bundle some trivial partition description for the
boards we have in board/ and reference it in the corresponding defconfigs.

How does 'genimages' work?

First, genimages extract the generated rootfs.tar. Hence, rootfs.tar is
always generated if a partition description is configured.

Second, genimages generates one filesystem image for each partition listed
in the description. Once the filesystem is generated, its mountpoint is
emptied, so it does not appear in the upper filesystem.

Third, genimages aggregates all the partition of a device into a single
image file

Finally, those device images can be flashed directly on the raw device of
the storage, without manually fiddling with  fdisk, mount, tar and so
on...

How does 'genimages' work internally?

The partition layout is decribed in a .ini-like file.

genimages is a relatively simple shell script that parses that .ini file.
Based on each device/partition description, it calls to 'handlers' (or
hooks), for each kind of content we support.

For example, we have two types of partitioning scheme supported for
device: mbr and gpt. Each is implemented in its own shell-fragment.
Ditto for partitions: we support different filesystems, ext, squashfs,
and vfat; they are each implemented in their own shell-fragment.

Each handler has to define the 'do_image' shell function. That functions
should expect two arguments: the base dir of the filesystem, and the
filename of the image to generate, and it has access to the configuration
from the .ini file.

How to use 'genimages'?

It's all explained in the manual. A pre-rendered version is available for
reading, and easy review on-line there:
    http://ymorin.is-a-geek.org/download/tmp/buildroot/manual/manual.html

The new section is "3.3.6 Customizing the generated filesystem images"
and there is a detailed description of the partition table layout in
appendix "12.3. Partition table layout description syntax".

Changes v2 -> v3:
  - misc typoes in the manual

Changes v1 -> v2:
  - drop rpi-firmware install into /boot  (Thomas, Ryan)
  - drop move of rpi-firmware in bootloader sub-menu  (Thomas)
  - add possibility to generate an unmounted filesystem  (Thomas, Ryan)
  - add ability to specify files to add in a filesystem instead of a
    sub-dir of target/  (Ryan, during a hacking session)
  - properly depend on the host-tools if they are selected
  - add GPT support
  - add a third filesystem, squashfs (in its own patch, as an example
    of how easy it is to add one)

Regards,
Yann E. MORIN.

The following changes since commit 2746158497cdaf80a5b85584e3972857ac2e7e82:

  util-linux: Add config switches for some more binaries (2013-12-20 22:45:31 +0100)

are available in the git repository at:

  git://gitorious.org/buildroot/buildroot.git yem/genimages

for you to fetch changes up to 81062cef5d4d51b6f196fb7c3d9c171c7c362555:

  fs/custom: add support for squashfs (2013-12-20 23:12:22 +0100)

----------------------------------------------------------------
Yann E. MORIN (4):
      fs/custom: generate complete, partition-based device images
      board/raspberrypi: provide partition description for the new genimages
      package/squashfs: add selection for the host variant
      fs/custom: add support for squashfs

 board/raspberrypi/partitions          |  36 ++++
 configs/raspberrypi_defconfig         |   8 +
 docs/manual/appendix.txt              |   1 +
 docs/manual/customize-filesystems.txt |  35 ++++
 docs/manual/customize.txt             |   2 +
 docs/manual/partition-layout.txt      | 323 ++++++++++++++++++++++++++++++++++
 fs/Config.in                          |   1 +
 fs/custom/Config.in                   |  16 ++
 fs/custom/boot/gpt                    | 126 +++++++++++++
 fs/custom/boot/mbr                    |  57 ++++++
 fs/custom/boot/pre-post               |   8 +
 fs/custom/custom.mk                   |  41 +++++
 fs/custom/fs/ext                      |  22 +++
 fs/custom/fs/pre-post                 |  67 +++++++
 fs/custom/fs/squashfs                 |  15 ++
 fs/custom/fs/vfat                     |  17 ++
 fs/custom/functions                   |  47 +++++
 fs/custom/genimages                   | 242 +++++++++++++++++++++++++
 package/Config.in.host                |   1 +
 package/squashfs/Config.in.host       |   6 +
 20 files changed, 1071 insertions(+)
 create mode 100644 board/raspberrypi/partitions
 create mode 100644 docs/manual/customize-filesystems.txt
 create mode 100644 docs/manual/partition-layout.txt
 create mode 100644 fs/custom/Config.in
 create mode 100644 fs/custom/boot/gpt
 create mode 100644 fs/custom/boot/mbr
 create mode 100644 fs/custom/boot/pre-post
 create mode 100644 fs/custom/custom.mk
 create mode 100644 fs/custom/fs/ext
 create mode 100644 fs/custom/fs/pre-post
 create mode 100644 fs/custom/fs/squashfs
 create mode 100644 fs/custom/fs/vfat
 create mode 100644 fs/custom/functions
 create mode 100755 fs/custom/genimages
 create mode 100644 package/squashfs/Config.in.host

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images
  2013-12-20 22:32 [Buildroot] [pull request v3] Pull request for branch yem/genimages Yann E. MORIN
@ 2013-12-20 22:32 ` Yann E. MORIN
  2013-12-21  8:49   ` Thomas Petazzoni
  2013-12-20 22:32 ` [Buildroot] [PATCH 2/4] board/raspberrypi: provide partition description for the new genimages Yann E. MORIN
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 11+ messages in thread
From: Yann E. MORIN @ 2013-12-20 22:32 UTC (permalink / raw)
  To: buildroot

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Contrary to the existing fs/ schemes, which each generate only a single
filesystem image for the root filesystem, this new scheme allows the
user to generate more complex images.

The basis behind this is a .ini-like description of the layout of the
final target storage:
  - the list of device(s)
  - per-device, the list of partition(s)
  - per-partition, the content

It is possible to create MBR- or GPT-based partitoining schemes. Adding
new ones should be relatively easy (but would need adequate host tools).

For now, the only content possible for partitions is a filesystem. It
should be pretty easy to add new types (eg. un-formated, or raw blob).

Also, only two filesystems are supported: ext{2,3,4} and vfat. Adding
more will be relatively easy, provided we have the necessary host
packages to deal with those filesystems.

The existing Buildroot filesystem generators are re-used as much as
possible when it makes sense; when it does not (eg. for vfat), a specific
generator is used.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ryan Barnett <rjbarnet@rockwellcollins.com>
---
 docs/manual/appendix.txt              |   1 +
 docs/manual/customize-filesystems.txt |  35 ++++
 docs/manual/customize.txt             |   2 +
 docs/manual/partition-layout.txt      | 314 ++++++++++++++++++++++++++++++++++
 fs/Config.in                          |   1 +
 fs/custom/Config.in                   |  16 ++
 fs/custom/boot/gpt                    | 126 ++++++++++++++
 fs/custom/boot/mbr                    |  57 ++++++
 fs/custom/boot/pre-post               |   8 +
 fs/custom/custom.mk                   |  38 ++++
 fs/custom/fs/ext                      |  22 +++
 fs/custom/fs/pre-post                 |  67 ++++++++
 fs/custom/fs/vfat                     |  17 ++
 fs/custom/functions                   |  47 +++++
 fs/custom/genimages                   | 242 ++++++++++++++++++++++++++
 15 files changed, 993 insertions(+)
 create mode 100644 docs/manual/customize-filesystems.txt
 create mode 100644 docs/manual/partition-layout.txt
 create mode 100644 fs/custom/Config.in
 create mode 100644 fs/custom/boot/gpt
 create mode 100644 fs/custom/boot/mbr
 create mode 100644 fs/custom/boot/pre-post
 create mode 100644 fs/custom/custom.mk
 create mode 100644 fs/custom/fs/ext
 create mode 100644 fs/custom/fs/pre-post
 create mode 100644 fs/custom/fs/vfat
 create mode 100644 fs/custom/functions
 create mode 100755 fs/custom/genimages

diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
index 74ee8fd..53f4205 100644
--- a/docs/manual/appendix.txt
+++ b/docs/manual/appendix.txt
@@ -6,6 +6,7 @@ Appendix
 
 include::makedev-syntax.txt[]
 include::makeusers-syntax.txt[]
+include::partition-layout.txt[]
 
 
 // Automatically generated lists:
diff --git a/docs/manual/customize-filesystems.txt b/docs/manual/customize-filesystems.txt
new file mode 100644
index 0000000..cddee42
--- /dev/null
+++ b/docs/manual/customize-filesystems.txt
@@ -0,0 +1,35 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[filesystem-custom]]
+Customizing the generated filesystem images
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
++Buildroot+ knows by default how to generate a few different kind of
+filesystems, such as +squashfs+, +ext2/3/4+, +cramfs+... But those
+filesystems are all generated to contain the complete target directory
+hierarchy in a single filesystem, mounted as the root filesystem +/+.
+That is, even if you select both an +ext2+ and a +squashfs+ filesystems,
+the content of the two generated images will be the exact same, only the
+types of the filesystems will be different.
+
+Most devices require a more complex setup, with different parts of the
+directory structure split across different filesystems, each stored on
+different partitions of one or more storage devices.
+
++Buildroot+ can generate such complex setups, using a +partition table layout
+description+. This is a simple text file, not unlike the +.ini+ style of
+configuration files, that describes how the target directory hierarchy has
+to be split across the target storage devices. It is a bit like a flattened
+tree of the storage layout.
+
+Set the variable +BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE+ to the path of
+the file containing your +partition table layout description+.
+
+See xref:part-layout-desc-syntax[] for the complete documentation of the
++partition table layout description+ syntax.
+
+[underline]*Note:* Although more versatile than the single filesystem image
+mechanism, the +partition table layout description+ might be unable to
+describe very complex setups. For example, it is not capable of handling
++initramfs+ based systems, or NFS-mounted filesystems.
diff --git a/docs/manual/customize.txt b/docs/manual/customize.txt
index 7e46fd8..6d062ea 100644
--- a/docs/manual/customize.txt
+++ b/docs/manual/customize.txt
@@ -14,6 +14,8 @@ include::customize-kernel-config.txt[]
 
 include::customize-toolchain.txt[]
 
+include::customize-filesystems.txt[]
+
 include::customize-store.txt[]
 
 include::customize-packages.txt[]
diff --git a/docs/manual/partition-layout.txt b/docs/manual/partition-layout.txt
new file mode 100644
index 0000000..553f862
--- /dev/null
+++ b/docs/manual/partition-layout.txt
@@ -0,0 +1,314 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+[[part-layout-desc-syntax]]
+
+Partition table layout description syntax
+-----------------------------------------
+
+The +partition table layout description+ syntax is not unlike the standard
+https://en.wikipedia.org/wiki/.ini[+.ini+] syntax. There are two types of
+entries: +sections+, that may each contain zero or more +properties+.
+
++Sections+ are specified between square brackets +[]+, _eg._: +[name]+.
+
++Properties+ are specified as key-value pairs, _eg._: +key=value+, and
+are documented as:
+
+* +key-name+ (optional or mandatory): description
+** +value1+: description
+** +value2+: description
+** ...
+
+[underline]*Note:* Unlike the standard +.ini+ syntax, the +partition table
+layout description+ _is_ case-sensitive.
+
+The order of +sections+ is irrelevant. However, for readability, we recomend
+the +partition table layout description+ starts with the +global+ section.
+
+The global section
+~~~~~~~~~~~~~~~~~~
+
+The +[global]+ section defines some global settings, and the list of devices.
+
+Properties for the global section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +extract+ (mandatory): the type of base image to extract
+** +tar+: extract the +rootfs.tar+ base image generated by +Buildroot+
+
+* +devices+ (mandatory): the comma-separated list of storage devices to
+  use on the device. Each device is the filename of the device node
+  present in +/dev+
+
+* +keep_parts+ (optional): also copy the individual partition images
+  of all devices to +$(BINARIES_DIR)+. Settings from the device sections
+  or the partition sections take precedence over this one.
+** +yes+: copy the individual partition images
+** +no+ (the default): do not copy individual partition images
+
+The devices and partitions sections
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The devices and partitions sections define, for each device or partition,
+the content of that device or partition.
+
+For each device listed in the +[global]+ section, there must be a
+corresponding section named after that device.
+
+For each partition listed in a device section, there must be a corresponding
+section named after that partition.
+
+Properties for the device section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +type+ (mandatory): the type of content for that device or partition
+** +boot+: the device contains one or more partitions, and
+   _may_ serve as a boot device
+
+* +keep_parts+ (optional): also copy the individual partition images
+  for this device to +$(BINARIES_DIR)+. Settings from the partition
+  sections take precedence over this one.
+** +yes+: copy the individual partition images
+** +no+ (the default): do not copy individual partition images
+
+Properties for the partition section
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +type+ (mandatory): the type of content for that device or partition
+** +fs+: the partition contains a filesystem
+
+* +size+: the size of that partition, in bytes
+
+* +keep+ (optional): copy this partition image to +$(BINARIES_DIR)+
+** +yes+: copy this partition image
+** +no+ (the default): do not copy this partition image
+
+Properties for +type=boot+
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +boot_type+ (mandatory): the partitioning scheme to use on this device
+** +mbr+: use an https://en.wikipedia.org/wiki/Master_boot_record[MBR]
+   partitioning scheme
+** +gpt+: use a https://en.wikipedia.org/wiki/GUID_Partition_Table[GPT]
+   partitioning scheme
+
+* +partitions+ (mandatory): the comma-separated list of partition(s) on
+  this device; no two partitions may have the same name, even if they
+  reside on different devices; partitions names shall match this regexp:
+  `^[[:alpha:]][[:alnum:]-_]*$` (_ie._ starts with a letter, followed by
+  zero or more alpha-numeric character or a dash or an underscore)
+
+* +partalign+ (optional): the alignment of partitions, in bytes; defaults
+  to an alignment of one, which means no alignment; depending on the
+  +boot_type+, some restrictions may apply, and are documented for each
+  +boot_type+
+
+Properties for +boot_type=mbr+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +mbr_bootcode+ (optional): the bootcode to use, as a path to the file
+  containing the bootcode image, relative to the +$(BINARIES_DIR)+
+  directory; defaults to no bootcode (eg. filled with zeroes)
+
+* +partalign+: must be a multiple of 512
+
+Properties for +boot_type=gpt+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +partalign+: must be a multiple of 512
+
+********
+Currently, only 512-byte sectors are supported. 4k sectors are not.
+********
+
+Properties for partitions whose containing device is +boot_type=mbr+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +mbr_type+ (mandatory): the partition
+  https://en.wikipedia.org/wiki/Partition_type#List_of_partition_IDs[type]
+
+Properties for +type=fs+
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +fs_type+ (mandatory): the type of filesystem to generate
+** +ext+: generate an extended filesystem (ext2, ext3, ext4)
+** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
+
+* +fs_label+ (optional): the label to assign to this filesystem, if that
+  filesystem supports a label
+
+* +fs_files_0+, +fs_files_1+, +fs_files_N+ (optional): the list of files,
+  relative to $(BINARIES_DIR), to store in the filesystem. These entries
+  must be indexed starting from 0, and must be sequential: the first
+  missing entry ends the list
+
+* +fs_root+ (optional): the mountpoint of the filesystem
+
+* +fs_vfstype+ (optional): the type of filesystem to use when calling
+  +mount+, if different from +fs_type+
+
+* +fs_mntopts+ (optional): the mount options; defaults to +defaults+
+
+_Note_: valid use-cases for +fs_root+ and +fs_files_N+:
+
+* if only +fs_root+ is specified (and no +fs_files_N+): the filesystem
+  content is made exclusively from +$(TARGET_DIR)/$(fs_root)+, and the
+  filesystem is mounted at runtime
+
+* if both +fs_root+ and at least one +fs_files_N+ are specified: the
+  filesystem content is made exclusively from the +fs_files_N+ entries,
+  and mounted at runtime. +$(TARGET_DIR)/$(fs_root)+ must be empty
+
+* if at least one +fs_files_N+ is specified, and +fs_root+ is not: the
+  filesystem content is made exclusively from the +fs_files_N+ entries,
+  and the filesystem is not mounted at runtime
+
+* if neither +fs_root+ nor +fs_files_N+ is specified: this is an error
+
+Properties for +fs_type=ext+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +ext_gen+ (mandatory): the generation of extended filesystem to generate
+** +2+, +3+, +4+: for an ext2, ext3 or ext4 filesystem
+
+* +ext_rev+ (mandatory): the revision of the extended filesystem
+** +0+ (ext2 only): generate a revision 0 extended filesystem filesystem
+** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
+   filesystem
+
+Properties for +fs_type=vfat+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +vfat_size+ (optional): the VFAT-size of the filesystem
+** +12+, +16+, +32+: generate a FAT12, FAT16, or FAT32
+
+Generation of the filesystems
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The filesystems are generated in an order such that a filesystem that is
+mounted as a sub-directory of another filesystem is generated first.
+
+A filesystem is filled with the content of the directory corresponding to
+its mountpoint, and then that directory is emptied before continuing to the
+next filesystem. That way, the mountpoints are empty before the filesystem
+that contains them are generated.
+
+Finally, an entry in added in +/etc/fstab+ for each generated filesystem, so
+they are mounted at boot time.
+
+Requirements on host packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Depending on what types of partitioning and filesystems are used by your
++partition table layout description+, you may have to enable some
+host-packages in the +Host utilities+ sub-menu.
+
+Since the content of a +partition table layout description+ is very
+specific to a board and/or a project, there is no way for Buildroot to
+automatically select those host packages, and thus it is your
+responsibility to select the appropriate ones.
+
+For example, for an MBR partitioning, you will have to enable the +host
+genpart+ package. For FAT filesystems, you will have to enable both of
++host dosfstools+ and +host mtools+.
+
+Examples
+~~~~~~~~
+
+.Simplest partition table layout description
+====
+----
+[global]
+extract=tar
+devices=sda
+
+[sda]
+type=boot
+boot_type=mbr
+partitions=root
+partalign=1048576
+
+[root]
+type=fs
+fs_type=ext
+fs_vfstype=ext4
+fs_root=/
+ext_gen=4
+ext_rev=1
+----
+
+The +partition table layout description+ above defines a single device
++sda+. That device contains a single partition, +root+, with an ext4
+filesystem, which is filled with the whole content of the +rootfs.tar+,
+and is mounted on +/+.
+====
+
+.More copmplex table layout description
+====
+----
+[global]
+extract=tar
+devices=mmcblk0,sda
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1024*1024))
+
+[sda]
+type=boot
+boot_type=mbr
+partitions=data
+partalign=4096
+
+[boot]
+type=fs
+mbr_type=$((0xC))
+size=$((16*1048576))
+fs_type=vfat
+fs_mntopts=ro
+fs_label=BOOT
+fs_root=/boot
+vfat_size=32
+
+[root]
+type=fs
+mbr_type=$((0x83))
+size=268435456
+fs_type=ext
+fs_vfstype=ext4
+fs_mntopts=discard,delalloc
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[data]
+type=fs
+mbr_type=$((0x83))
+size=$((4*1024*1048576))
+fs_type=ext
+fs_vfstype=ext2
+fs_root=/data
+fs_label=DATA
+ext_gen=2
+ext_rev=1
+----
+====
+
+The example above defines two devices, +mmcblk0+ and +sda+.
+
+The +mmcblk0+ device contains two partitions, +boot+ and +root+; partitions
+are aligned on a 1MiB boundary. The +sda+ device contains a single partition,
++data+, aligned on a 4KiB boundary.
+
+The +boot+ partition is a 16MiB FAT32 filesystem filled with the content
+of, and mounted on, +/boot+, and with label +BOOT+.
+
+The +data+ partition is a 4GiB ext2r1 filesystem filled with the content
+of, and mounted on, +/data+, and with label +DATA+.
+
+The +root+ partition is a 256MiB ext4 filesystem filled the the rest of,
+and mounted on, +/+, and with label +ROOT+.
diff --git a/fs/Config.in b/fs/Config.in
index da4c5ff..44e04f7 100644
--- a/fs/Config.in
+++ b/fs/Config.in
@@ -3,6 +3,7 @@ menu "Filesystem images"
 source "fs/cloop/Config.in"
 source "fs/cpio/Config.in"
 source "fs/cramfs/Config.in"
+source "fs/custom/Config.in"
 source "fs/ext2/Config.in"
 source "fs/initramfs/Config.in"
 source "fs/iso9660/Config.in"
diff --git a/fs/custom/Config.in b/fs/custom/Config.in
new file mode 100644
index 0000000..e5a8ee7
--- /dev/null
+++ b/fs/custom/Config.in
@@ -0,0 +1,16 @@
+config BR2_TARGET_ROOTFS_CUSTOM
+	bool "Custom partition table layout"
+	select BR2_TARGET_ROOTFS_TAR
+
+config BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE
+	string "path to the custom partition table layout description"
+	depends on BR2_TARGET_ROOTFS_CUSTOM
+	help
+	  Enter the path to a partition-table for your device.
+
+	  This will allow Buildroot to generate a more complex target
+	  image, which may consist of more than one filesystem on more
+	  than one partition.
+
+	  See docs/manual/bla-bla on how to construct such a partition
+	  table.
diff --git a/fs/custom/boot/gpt b/fs/custom/boot/gpt
new file mode 100644
index 0000000..8592cbf
--- /dev/null
+++ b/fs/custom/boot/gpt
@@ -0,0 +1,126 @@
+# Build a complete GPT-based image
+
+# For a GPT-based partitionning, we need to compute the complete
+# image size before we can attempt to generate the partition table.
+# Then, we need to add the size of the GPT itself, plus that of its
+# backup copy, plus the protective MBR.
+# The size of the GPT itself depends on the sector size, and the
+# number of partitions in the GPT. Sectors can be either 512-byte
+# or 4096-byte large; The numbers of partitions is unlimited, but
+# it is suggested there is space to store at least 128 of them; a
+# partition description is 128-byte large.
+#
+# https://en.wikipedia.org/wiki/GUID_Partition_Table
+#
+# So, here's what we do:
+#   - consider 512-byte sectors (since GPT on 4k sectors is not well
+#     documented)
+#   - consider at least 128 partitions; if the layout defines more than
+#     that, we need to round that number up to the smallest multiple of
+#     4 (since there are 4 partition descriptions in a 512-byte sector)
+#   - generate an empty, sparse file that is big enough to store the MBR,
+#     the two GPT copies, and the aligned partitions.
+#   - dump each partition in turn in their final location in that file
+#   - generate a parted script that creates the partition table in that
+#     file
+#
+#   Simg = 512 + 2*(Sgpt) + ?( aligned(Spart,512) )
+#   Sgpt = 512 + Nent*128
+#
+# Where:
+#   Simg        : size of the image
+#   Sgpt        : size of one GPT
+#   Spart       : size of each partition
+#   Nent        : number of partition entries
+#   aligned()   : the alignment function
+#
+# Sicne 4k-large sectors are not really explained on Wikipedia, we can
+# add this later on.
+
+do_image() {
+    # ${1} is fs_root, irrelevant here
+    local img="${2}"
+
+    # How many partitions do we have?
+    nb_parts=0
+    for part in ${partitions[${dev}]//,/ }; do
+        nb_parts=$((nb_parts+1))
+    done
+
+    # How many partition entries do we need?
+    nb_entries=$((4*((nb_parts+3)/4)))
+    nb_entries=$((nb_entries<128?128:nb_entries))
+
+    # The size of a single GPT
+    gpt_size=$((512+(128*nb_entries)))
+
+    # Offset of the first partition
+    begin=$(align_val $((512+$(align_val ${gpt_size} 512))) ${partalign} )
+
+    # Initialise our image file
+    dd if=/dev/zero of="${img}"     \
+       bs=1 seek=${begin} count=0   \
+       conv=sparse                  2>/dev/null
+
+    # Compute the space required to store all partitions
+    # and store them in the image file
+    size_parts=0
+    _offset=${begin}
+    i=1
+    debug "adding partions descriptions\n"
+    for part in ${partitions[${dev}]//,/ }; do
+        debug "  %s\n" "${part}"
+        _part_img="${tmp_dir}/${dev}.${part}.img"
+        _size=$( align_val $( stat -c '%s' "${_part_img}" ) 512 )
+        part_offset+=( ${_offset} )
+        _attr="${values["${part}:gpt_attr"]}"
+        _label="${values["${part}:gpt_label"]}"
+
+        # If the partition has no label, use the filesystem label
+        if [ -z "${_label}" ]; then
+            _label="${values["${part}:fs_label"]}"
+        fi
+        if [ -z "${_label}" ]; then
+            _label="data"
+        fi
+
+        debug "    start=%s\n" "${_offset}"
+        debug "    size =%s\n" "${_size}"
+        debug "    end  =%s\n" "$((_offset+_size-1))"
+
+        dd if="${_part_img}" of="${img}"    \
+           bs=512 seek=$((_offset/512))     \
+           conv=notrunc,sparse              2>/dev/null
+
+        parted_script+=( mkpart "${_label}"    \
+                                ${_offset}              \
+                                $((_offset+_size-1))    \
+                       )
+        if [ -n "${_attr}" ]; then
+            for attr in "${_attr//,/ }"; do
+                parted_script+=( set ${i} ${attr} on )
+            done
+        fi
+
+        size_parts=$((size_parts+_size))
+        _offset=$((_offset+_size))
+        i=$((i+1))
+    done
+
+    # Terminate our image file
+    img_size=$(align_val $(( begin + size_parts + gpt_size )) 512)
+    debug "begin   =%s\n" ${begin}
+    debug "nb_entry=%s\n" ${nb_entries}
+    debug "gpt_size=%s\n" ${gpt_size}
+    debug "img_size=%s\n" ${img_size}
+    dd if=/dev/zero of="${img}" \
+       bs=1 seek=${img_size}    \
+       count=0 conv=sparse      2>/dev/null
+
+    for i in parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"; do
+        debug "--> '%s'\n" "${i}"
+    done
+    parted -s "${img}" mklabel gpt unit B "${parted_script[@]}"
+}
+
+# vim: ft=sh
diff --git a/fs/custom/boot/mbr b/fs/custom/boot/mbr
new file mode 100644
index 0000000..af7c6cb
--- /dev/null
+++ b/fs/custom/boot/mbr
@@ -0,0 +1,57 @@
+# Build a complete MBR-based image
+
+do_image() {
+    # ${1} is fs_root, irrelevant here
+    local img="${2}"
+    local i begin part part_img size type _begin _size
+    local -a part_offset part_file
+
+    # Fill-in the boot record with zeroes
+    # Ideally, we should dump the bootloader code here, but since
+    # we don't have any so far, that will have to be done in a
+    # later step.
+    debug "adding (fake) bootcode\n"
+    dd if=/dev/zero of="${img}" bs=$((0x1be)) count=0 seek=1 2>/dev/null
+
+    # Generate partition entries
+    i=0
+    begin=${partalign}
+    debug "adding partitions descriptors\n"
+    for part in ${partitions[${dev}]//,/ }; do
+        debug "  %s\n" "${part}"
+        part_offset+=( ${begin} )
+        part_img="${tmp_dir}/${dev}.${part}.img"
+        part_file+=( "${part_img}" )
+        size=$( align_val $( stat -c '%s' "${part_img}" ) 512 )
+        type="${values["${part}:mbr_type"]}"
+        # LBA is exressed in a number of 512-byte blocks
+        # and genparts only deals with LBA
+        _begin=$((begin/512))   # begin is already 512-byte aligned
+        _size=$((size/512))     # size is already 512-byte aligned
+        debug "    start=%s (LBA %s)\n" "${begin}" "${_begin}"
+        debug "    size =%s (LBA %s)\n" "${size}"  "${_size}"
+        debug "    type =%s\n"          "${type}"
+        genpart -b ${_begin} -s ${_size} -t ${type} >>"${img}"
+        begin=$( align_val $((begin+size)) ${partalign} )
+        i=$((i+1))
+    done
+    nb_parts=${i}
+    # Generate entries for empty partitions
+    for(( ; i<4; i++ )); do
+        debug "  (empty)\n"
+        genpart -t 0 >>"${img}"
+    done
+    # Dump the boot signature
+    printf "\x55\xaa" >>"${img}"
+
+    for(( i=0; i<nb_parts; i++ )); do
+        part_img="${part_file[${i}]}"
+        offset=${part_offset[${i}]}
+        _offset=$(( offset/512 ))  # offset is already 512-byte aligned
+        dd if="${part_img}" of="${img}" \
+           bs=512 seek=${_offset}       \
+           conv=notrunc,sparse          2>/dev/null
+    done
+}
+
+# vim: ft=sh
diff --git a/fs/custom/boot/pre-post b/fs/custom/boot/pre-post
new file mode 100644
index 0000000..507d17e
--- /dev/null
+++ b/fs/custom/boot/pre-post
@@ -0,0 +1,8 @@
+do_image_pre() {
+    :
+}
+do_image_post() {
+    :
+}
+
+#vim: set ft=sh
diff --git a/fs/custom/custom.mk b/fs/custom/custom.mk
new file mode 100644
index 0000000..63c1f23
--- /dev/null
+++ b/fs/custom/custom.mk
@@ -0,0 +1,38 @@
+################################################################################
+#
+# custom partitioning
+#
+################################################################################
+
+define ROOTFS_CUSTOM_CMD
+	BUILD_DIR=$(BUILD_DIR) fs/custom/genimages \
+		'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
+endef
+
+# rootfs-custom uses rootfs.tar as the source to generate
+# the resulting image(s), so we need to build it first.
+ROOTFS_CUSTOM_DEPENDENCIES += rootfs-tar
+
+# All of the following filesystem generators, or partition managers, are
+# optional, but if they are selected, we may need them, so we need to
+# depend on them
+ifeq ($(BR2_PACKAGE_HOST_DOSFSTOOLS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-dosfstools
+endif
+ifeq ($(BR2_PACKAGE_HOST_E2FSPROGS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-e2fsprogs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENEXT2FS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-genext2fs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENPART),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-genpart
+endif
+ifeq ($(BR2_PACKAGE_HOST_MTOOLS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-mtools
+endif
+ifeq ($(BR2_PACKAGE_HOST_PARTED),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-parted
+endif
+
+$(eval $(call ROOTFS_TARGET,custom))
diff --git a/fs/custom/fs/ext b/fs/custom/fs/ext
new file mode 100644
index 0000000..5f9b4e7
--- /dev/null
+++ b/fs/custom/fs/ext
@@ -0,0 +1,22 @@
+# Create an extended file system
+
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+    local gen rev
+
+    fs_opts+=( -z )
+    fs_opts+=( -d "${root_dir}" )
+    [ -z "${size}"    ] || fs_opts+=( -b $((size/1024)) )
+    [ -n "${ext_gen}" ] || ext_gen=2
+    [ -n "${ext_rev}" ] || ext_rev=1
+
+    # Remember, we're running from Buildroot's TOP_DIR
+    GEN=${ext_gen} REV=${ext_rev}                   \
+    ./fs/ext2/genext2fs.sh "${fs_opts[@]}" "${img}" >/dev/null
+
+    [ -z "${fs_label}" ] || tune2fs -L "${fs_label}" "${img}" >/dev/null
+}
+
+# vim: ft=sh
diff --git a/fs/custom/fs/pre-post b/fs/custom/fs/pre-post
new file mode 100644
index 0000000..40ec047
--- /dev/null
+++ b/fs/custom/fs/pre-post
@@ -0,0 +1,67 @@
+#-----------------------------------------------------------------------------
+do_image_pre() {
+    local i file
+
+    # if fs_root_dir is not specified, we have to create one
+    # It *does* override the caller's fs_root_dir value, but
+    # that's on purpose
+    # If fs_root_dir is specified, and we have at least fs_files_0,
+    # then fs_root_dir/ must be enpty
+    if [ -z "${fs_root_dir}" ]; then
+        if [ -n "${values["${part}:fs_files_0"]}" ]; then
+            error "%s: no fs_root specified, and no fs_files_0\n" "${part}"
+        fi
+        fs_root_dir="$( mktemp -d "${tmp_dir}/XXXXXX" )"
+    else
+        if [    -n "${values["${part}:fs_files_0"]}"      \
+             -a $( ls -1A "${fs_root_dir}" |wc -l ) -ne 0 ]; then
+            error "%s: %s is not empty, but fs_files_0 is specified\n"  \
+                  "${part}" "${fs_root_dir#${fs_root}}"
+        fi
+    fi
+
+    i=0
+    while true; do
+        file="${values["${part}:fs_files_${i}"]}"
+        [ -n "${file}" ] || break
+        debug "%s: adding fs_files_%d %s\n" "${part}" ${i} "${file}"
+        install -D "${BINARIES_DIR}/${file}" "${fs_root_dir}/${file##*/}"
+        i=$((i+1))
+    done
+}
+
+#-----------------------------------------------------------------------------
+do_image_post() {
+    local rootfs_dir="${1}"
+    local fs_root="${2}"
+    local img_file="${3}"
+    local part="${4}"
+    local dev mntops vfstype fs_root_esc
+
+    subname+="[post-image]"
+
+    # Empty the partition's mountpoint
+    find "${fs_root_dir}" -maxdepth 1 \! -path "${fs_root_dir}" -exec rm -rf {} +
+
+    # Add entry in fstab, but not if this is '/'
+    # Don't add either if rootfs was not extracted
+    if [    "${fs_root}" = "/" -o -z "${fs_root}" \
+         -o -z "${values["global:extract"]}"      ]; then
+        return 0
+    fi
+    fs_root_esc="$( sed -r -e 's:/:\\/:g;' <<<"${fs_root}" )"
+    sed -r -i -e "/[^[:space:]]+[[:space:]]+${fs_root_esc}[[:space:]]/d"    \
+                 "${rootfs_dir}/etc/fstab"
+    dev="$( get_part_dev_node "${part}" )"
+    vfstype="${fs_vfstype:-${fs_type}}"
+    mntops="${fs_mntops:-defaults}"
+    printf "/dev/%s %s %s %s 0 0\n"     \
+           "${dev}" "${fs_root}"        \
+           "${vfstype}" "${mntops}"     \
+           >>"${rootfs_dir}/etc/fstab"
+
+    subname="${subname%\[post-image\]}"
+}
+
+#-----------------------------------------------------------------------------
+# vim: ft=sh
diff --git a/fs/custom/fs/vfat b/fs/custom/fs/vfat
new file mode 100644
index 0000000..aa5545f
--- /dev/null
+++ b/fs/custom/fs/vfat
@@ -0,0 +1,17 @@
+# Create a VFAT file system
+
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+
+    dd if=/dev/zero of="${img}" bs=${size} count=0 seek=1 2>/dev/null
+
+    [ -z "${vfat_size}" ] || fs_opts+=( -F ${vfat_size} )
+    [ -z "${fs_label}"  ] || fs_opts+=( -n "${fs_label}" )
+    mkfs.vfat "${fs_opts[@]}" "${img}" >/dev/null
+
+    mcopy -i "${img}" "${root_dir}/"* '::'
+}
+
+# vim: ft=sh
diff --git a/fs/custom/functions b/fs/custom/functions
new file mode 100644
index 0000000..4b41021
--- /dev/null
+++ b/fs/custom/functions
@@ -0,0 +1,47 @@
+# Common functions
+
+#------------------------------------------------------------------------------
+align_val() {
+    local val="${1}"
+    local align="${2}"
+    local aligned
+
+    aligned=$(( ( (val+align-1) / align ) * align ))
+
+    printf "%d" ${aligned}
+}
+
+#------------------------------------------------------------------------------
+# Some trace functions
+trace() {
+    local fmt="${1}"
+    shift
+
+    printf "%s" "${myname}"
+    if [ -n "${subname}" ]; then
+        printf "(%s)" "${subname}"
+    fi
+    printf ": ${fmt}" "${@}"
+}
+
+debug() { :; }
+if [ -n "${DEBUG}" ]; then
+    debug() {
+        trace "${@}" >&2
+    }
+fi
+
+error() {
+    trace "${@}" >&2
+    exit 1
+}
+
+on_error() {
+    local ret=${?}
+
+    error "unexpected error caught: %d\n" ${ret}
+}
+trap on_error ERR
+set -E -e
+
+# vim: ft=sh
diff --git a/fs/custom/genimages b/fs/custom/genimages
new file mode 100755
index 0000000..00820fc
--- /dev/null
+++ b/fs/custom/genimages
@@ -0,0 +1,242 @@
+#!/bin/bash
+set -e
+set -E
+
+#-----------------------------------------------------------------------------
+main() {
+    local part_table="${1}"
+    local tmp_dir
+    local rootfs_dir
+    local -a devices
+    local extract
+    local cur_section
+    local -a sections devices partitions
+    local -A variables values partdevs
+    local sec dev part var val
+    local secs devs parts vars vals
+    local has_global_section
+
+    # We need bash 4 or above for asociative arrays
+    if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
+        error "bash 4 or above is needed\n"
+    fi
+
+    if [ ! -f "${part_table}" ]; then
+        error "%s: no such file\n" "${part_table}"
+        exit 1
+    fi
+
+    export PATH="${HOST_DIR}/usr/bin:${HOST_DIR}/usr/sbin:${PATH}"
+
+    # Parse all the sections in one go, we'll sort
+    # all the mess afterwards...
+    debug "parsing partitions descriptions file '%s'\n" \
+          "${part_table}"
+    has_global_section=0
+    while read line; do
+        line="$( sed -r -e 's/[[:space:]]*#.*$//;' <<<"${line}" )"
+
+        # Detect start of global section, skip anything else
+        case "${line}" in
+        "") continue ;;
+        '[global]') has_global_section=1 ;;&
+        '['*']')
+            cur_section="$( sed -r -e 's/[][]//g;' <<<"${line}" )"
+            debug "  entering section '%s'\n" "${cur_section}"
+            sections+=( "${cur_section}" )
+            continue
+        ;;
+        ?*=*) ;;
+        *)  error "malformed entry '%s'\n" "${line}" ;;
+        esac
+
+        var="${line%%=*}"
+        eval val="${line#*=}"
+        debug "    adding '%s'='%s'\n" "${var}" "${val}"
+        variables+=( ["${cur_section}"]=",${var}" )
+        values+=( ["${cur_section}:${var}"]="${val}" )
+    done <"${part_table}"
+
+    if [ ${has_global_section} -eq 0 ]; then
+        error "no global section defined\n"
+    fi
+
+    # Create lists of devices, partitions, and partition:device pairs.
+    debug "creating intermediate lists\n"
+    devices=( ${values["global:devices"]//,/ } )
+    for dev in "${devices[@]}"; do
+        # Sanity check first: all devices must have a corresponding
+        # section, which means they should have a type
+        if [ -z "${values["${dev}:type"]}" ]; then
+            error "device '%s' has no type (no section?)\n" "${dev}"
+        fi
+        partitions+=( ${values["${dev}:partitions"]//,/ } )
+        for part in ${values["${dev}:partitions"]//,/ }; do
+            # Sanity check first: all partitions must have a corresponding
+            # section, which means they should have a type
+            if [ -z "${values["${part}:type"]}" ]; then
+                error "partition '%s' has no type (no section?)\n" "${dev}"
+            fi
+            partdevs+=( ["${part}"]="${dev}" )
+        done
+    done
+
+    # Now, we must order the partitions so that their mountpoint
+    # is empty by the time we build the upper-level partition.
+    # For example, given this layout of mountpoints:
+    #   /
+    #   /usr
+    #   /usr/var
+    # We must ensure /usr/var is empty@the time we create the /usr
+    # filesystem image; and similarly, we must ensure /usr is empty by
+    # the time we create the / filesystem image
+    # So, a simple reverse alphabetical sort will do the trick
+    debug "sorting partitions\n"
+    sorted_parts=( $(
+        for part in "${partitions[@]}"; do
+            # Partitions that are not mounted can be generated
+            # in any order
+            if [ -n "${values["${part}:fs_root"]}" ]; then
+                printf "%s:%s\n" "${part}" "${values["${part}:fs_root"]}"
+            else
+                printf "%s\n" "${part}"
+            fi
+        done                    \
+        |sort -t: -k2 -r        \
+        |sed -r -e 's/:[^:]+$//;'
+    ) )
+
+    tmp_dir="${BUILD_DIR}/genimages.tmp"
+    rootfs_dir="${tmp_dir}/rootfs"
+    # Since we don't remove it in case of error (to be able to inspect its
+    # content), we must remove it now (a previous run may have left it).
+    rm -rf "${tmp_dir}"
+    mkdir -p "${rootfs_dir}"
+
+    case "${values["global:extract"]}" in
+        tar)
+            trace "extracting rootfs.tar\n"
+            tar xf "${BINARIES_DIR}/rootfs.tar" -C "${rootfs_dir}"
+        ;;
+        *)  error "unknown extract method '%s'\n" "${extract:-(none)}"
+        ;;
+    esac
+
+    # Render all partition images
+    for part in "${sorted_parts[@]}"; do
+        trace "preparing filesystem for partition '%s'\n" "${part}"
+        render_img "${rootfs_dir}" "${part}"                        \
+                   "${tmp_dir}/${partdevs["${part}"]}.${part}.img"
+    done
+
+    # Aggregate all devices images
+    for dev in "${devices[@]}"; do
+        trace "assembling partitions in device '%s'\n" "${dev}"
+        render_img "${rootfs_dir}" "${dev}" "${tmp_dir}/${dev}.img"
+    done
+
+    # Copy all partitions and devices images to the image dir
+    if [ "${values["global:keep_partitions"]}" = "yes" ]; then
+        for part in "${sorted_parts[@]}"; do
+            debug "copying partition '%s' to image dir\n" "${part}"
+            dd if="${tmp_dir}/${partdevs["${part}"]}.${part}.img"           \
+               of="${BINARIES_DIR}/$( get_part_dev_node "${part}" ).img"    \
+               bs=4096 conv=sparse                                          2>/dev/null
+        done
+    fi
+    for dev in "${devices[@]}"; do
+        debug "copying device '%s' to image dir\n" "${dev}"
+        dd if="${tmp_dir}/${dev}.img"       \
+           of="${BINARIES_DIR}/${dev}.img"  \
+           bs=4096 conv=sparse              2>/dev/null
+    done
+
+    [ -n "${DEBUG}" ] || rm -rf "${tmp_dir}"
+}
+
+#-----------------------------------------------------------------------------
+render_img() {
+    local rootfs_dir="${1}"
+    local img="${2}"
+    local img_file="${3}"
+    local type sub_type fs_root_dir
+
+    type="${values["${img}:type"]}"
+    sub_type="${values["${img}:${type}_type"]}"
+
+    # Sanity checks
+    [ -n "${type}" ] || error "'%s': unspecified type\n" "${img}"
+    if [ ! -d "fs/custom/${type}" ]; then
+        error "'%s': unsupported type '%s'\n" "${img}" "${type}"
+    fi
+    [ -n "${sub_type}" ] || error "'%s': unspecified %s_type\n" "${img}" "${type}"
+    if [ ! -f "fs/custom/${type}/${sub_type}" ]; then
+        error "'%s': unknown %s_type '%s'\n" "${img}" "${type}" "${sub_type}"
+    fi
+
+    # Need to call the renderer in a subshell so that its definitions
+    # do not pollute our environment
+    subname="${sub_type}"
+    (
+        trap 'exit $?' ERR
+        for var in ${variables["${img}"]//,/ }; do
+            eval "${var}=\"${values["${img}:${var}"]}\""
+        done
+        fs_root_dir="${rootfs_dir}${fs_root}"
+        . "fs/custom/${type}/pre-post"
+        . "fs/custom/${type}/${sub_type}"
+        do_image_pre "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
+        do_image "${fs_root_dir}" "${img_file}"
+        do_image_post "${rootfs_dir}" "${fs_root}" "${img_file}" "${img}"
+    )
+    ret=${?}
+    [ ${ret} -eq 0 ] || exit ${ret}
+    subname=""
+}
+
+#-----------------------------------------------------------------------------
+get_part_dev_node() {
+    local part="${1}"
+    local dev
+    local i c p
+
+    dev="${partdevs["${part}"]}"
+    i="${values["${dev}:partstart"]:-1}"
+
+    # If device node ends with a number, partitions are denoted
+    # with a 'p' before the partition number, eg.:
+    #   /dev/mmcblk0    --> /dev/mmcblk0p1
+    #   /dev/sda        --> /dev/sda1
+    case "${dev#${dev%?}}" in
+        [0-9])  c="p";;
+        *)      c="";;
+    esac
+
+    for p in ${values["${dev}:partitions"]//,/ }; do
+        if [ "${p}" = "${part}" ]; then
+            printf "%s%s%d" "${dev}" "${c}" ${i}
+            return 0
+        fi
+        i=$((i+1))
+    done
+
+    error "'%s': partition not found. WTF?\n" "${part}"
+}
+
+#-----------------------------------------------------------------------------
+myname="${0##*/}"
+mydir="${0%/*}"
+
+TOP_DIR="$( pwd )"
+export myname mydir TOP_DIR
+
+. "fs/custom/functions"
+
+# This script can deal with extracting the rootfs tarball, but we need to
+# be root for that.
+if [ $(id -u) -ne 0 ]; then
+    printf "error: not root\n"
+    exit 1
+else
+    main "${@}"
+fi
-- 
1.8.1.2

^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 2/4] board/raspberrypi: provide partition description for the new genimages
  2013-12-20 22:32 [Buildroot] [pull request v3] Pull request for branch yem/genimages Yann E. MORIN
  2013-12-20 22:32 ` [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images Yann E. MORIN
@ 2013-12-20 22:32 ` Yann E. MORIN
  2013-12-20 22:32 ` [Buildroot] [PATCH 3/4] package/squashfs: add selection for the host variant Yann E. MORIN
  2013-12-20 22:32 ` [Buildroot] [PATCH 4/4] fs/custom: add support for squashfs Yann E. MORIN
  3 siblings, 0 replies; 11+ messages in thread
From: Yann E. MORIN @ 2013-12-20 22:32 UTC (permalink / raw)
  To: buildroot

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Now we can generate a complete target storage image with the genimages
infra, add a partition table layout description for the Raspberry Pi
as an example for how to use genimages.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ryan Barnett <rjbarnet@rockwellcollins.com>
---
 board/raspberrypi/partitions  | 36 ++++++++++++++++++++++++++++++++++++
 configs/raspberrypi_defconfig |  8 ++++++++
 2 files changed, 44 insertions(+)
 create mode 100644 board/raspberrypi/partitions

diff --git a/board/raspberrypi/partitions b/board/raspberrypi/partitions
new file mode 100644
index 0000000..c5e8de5
--- /dev/null
+++ b/board/raspberrypi/partitions
@@ -0,0 +1,36 @@
+[global]
+extract=tar
+devices=mmcblk0
+
+[mmcblk0]
+type=boot
+boot_type=mbr
+partitions=boot,root
+partalign=$((1048576))
+
+[root]
+type=fs
+size=$((32*1048576))
+mbr_type=$((0x83))
+fs_type=ext
+fs_vfstype=ext4
+fs_mntops=discard
+fs_root=/
+fs_label=ROOT
+ext_gen=4
+ext_rev=1
+
+[boot]
+type=fs
+# FAT32 is not possible under 9MiB
+size=$((9*1048576))
+mbr_type=$((0xc))
+fs_type=vfat
+fs_files_0=rpi-firmware/bootcode.bin
+fs_files_1=rpi-firmware/start.elf
+fs_files_2=rpi-firmware/fixup.dat
+fs_files_3=rpi-firmware/config.txt
+fs_files_4=rpi-firmware/cmdline.txt
+fs_files_5=zImage
+fs_label=BOOT
+vfat_size=16
diff --git a/configs/raspberrypi_defconfig b/configs/raspberrypi_defconfig
index 4df0f4b..c571817 100644
--- a/configs/raspberrypi_defconfig
+++ b/configs/raspberrypi_defconfig
@@ -21,3 +21,11 @@ BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="5ebf1db80e2c03e92d570605b53f91a06112574f"
 BR2_LINUX_KERNEL_USE_DEFCONFIG=y
 BR2_LINUX_KERNEL_DEFCONFIG="bcmrpi_quick"
 BR2_LINUX_KERNEL_ZIMAGE=y
+
+BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE="$(TOPDIR)/board/raspberrypi/partitions"
+
+BR2_PACKAGE_HOST_DOSFSTOOLS=y
+BR2_PACKAGE_HOST_E2FSPROGS=y
+BR2_PACKAGE_HOST_GENEXT2FS=y
+BR2_PACKAGE_HOST_GENPART=y
+BR2_PACKAGE_HOST_MTOOLS=y
-- 
1.8.1.2

^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 3/4] package/squashfs: add selection for the host variant
  2013-12-20 22:32 [Buildroot] [pull request v3] Pull request for branch yem/genimages Yann E. MORIN
  2013-12-20 22:32 ` [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images Yann E. MORIN
  2013-12-20 22:32 ` [Buildroot] [PATCH 2/4] board/raspberrypi: provide partition description for the new genimages Yann E. MORIN
@ 2013-12-20 22:32 ` Yann E. MORIN
  2013-12-20 22:32 ` [Buildroot] [PATCH 4/4] fs/custom: add support for squashfs Yann E. MORIN
  3 siblings, 0 replies; 11+ messages in thread
From: Yann E. MORIN @ 2013-12-20 22:32 UTC (permalink / raw)
  To: buildroot

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

If any of the post-image scripts, or genimages, wants to handle squashfs
filesystems, we need to expose an option for squashfs-tools to be
user-selectable.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ryan Barnett <rjbarnet@rockwellcollins.com>
---
 package/Config.in.host          | 1 +
 package/squashfs/Config.in.host | 6 ++++++
 2 files changed, 7 insertions(+)
 create mode 100644 package/squashfs/Config.in.host

diff --git a/package/Config.in.host b/package/Config.in.host
index 34e84bf..71597bf 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -12,6 +12,7 @@ source "package/omap-u-boot-utils/Config.in.host"
 source "package/openocd/Config.in.host"
 source "package/parted/Config.in.host"
 source "package/sam-ba/Config.in.host"
+source "package/squashfs/Config.in.host"
 source "package/sunxi-tools/Config.in.host"
 source "package/uboot-tools/Config.in.host"
 
diff --git a/package/squashfs/Config.in.host b/package/squashfs/Config.in.host
new file mode 100644
index 0000000..b353374
--- /dev/null
+++ b/package/squashfs/Config.in.host
@@ -0,0 +1,6 @@
+config BR2_PACKAGE_HOST_SQUASHFS
+	bool "host squashfs"
+	help
+	  Tools to generate SquashFS filesystems.
+
+	  http://squashfs.sourceforge.net/
-- 
1.8.1.2

^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 4/4] fs/custom: add support for squashfs
  2013-12-20 22:32 [Buildroot] [pull request v3] Pull request for branch yem/genimages Yann E. MORIN
                   ` (2 preceding siblings ...)
  2013-12-20 22:32 ` [Buildroot] [PATCH 3/4] package/squashfs: add selection for the host variant Yann E. MORIN
@ 2013-12-20 22:32 ` Yann E. MORIN
  3 siblings, 0 replies; 11+ messages in thread
From: Yann E. MORIN @ 2013-12-20 22:32 UTC (permalink / raw)
  To: buildroot

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ryan Barnett <rjbarnet@rockwellcollins.com>
---
 docs/manual/partition-layout.txt | 11 ++++++++++-
 fs/custom/custom.mk              |  3 +++
 fs/custom/fs/squashfs            | 15 +++++++++++++++
 3 files changed, 28 insertions(+), 1 deletion(-)
 create mode 100644 fs/custom/fs/squashfs

diff --git a/docs/manual/partition-layout.txt b/docs/manual/partition-layout.txt
index 553f862..7b64a45 100644
--- a/docs/manual/partition-layout.txt
+++ b/docs/manual/partition-layout.txt
@@ -133,7 +133,8 @@ Properties for +type=fs+
 
 * +fs_type+ (mandatory): the type of filesystem to generate
 ** +ext+: generate an extended filesystem (ext2, ext3, ext4)
-** +vfat+: generate a VFAT filesystem (FAT16, FAT32)
+** +squashfs+: generate a squashfs (version 4) filesystem
+** +vfat+: generate a VFAT filesystem (FAT12, FAT16 or FAT32)
 
 * +fs_label+ (optional): the label to assign to this filesystem, if that
   filesystem supports a label
@@ -177,6 +178,14 @@ Properties for +fs_type=ext+
 ** +1+ (mandatory for ext3 or ext4): generate a revision 1 extended
    filesystem
 
+Properties for +fs_type=squashfs+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* +squashfs_comp+ (mandatory): the compression type to use
+** +gzip+, +xz+, +lzo+: gzip, xz or lzo compression
+
+* +squashfs_block+ (optional): the size of blocks, in bytes
+
 Properties for +fs_type=vfat+
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/fs/custom/custom.mk b/fs/custom/custom.mk
index 63c1f23..158e886 100644
--- a/fs/custom/custom.mk
+++ b/fs/custom/custom.mk
@@ -34,5 +34,8 @@ endif
 ifeq ($(BR2_PACKAGE_HOST_PARTED),y)
 ROOTFS_CUSTOM_DEPENDENCIES += host-parted
 endif
+ifeq ($(BR2_PACKAGE_HOST_SQUASHFS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-squashfs
+endif
 
 $(eval $(call ROOTFS_TARGET,custom))
diff --git a/fs/custom/fs/squashfs b/fs/custom/fs/squashfs
new file mode 100644
index 0000000..ba3646e
--- /dev/null
+++ b/fs/custom/fs/squashfs
@@ -0,0 +1,15 @@
+# Create a squashfs filesystem
+
+do_image() {
+    local root_dir="${1}"
+    local img="${2}"
+    local -a fs_opts
+
+    fs_opts+=( -no-progress -noappend )
+    [ -z "${squashfs_comp}"  ] || fs_opts+=( -comp "${squashfs_comp}" )
+    [ -z "${squashfs_block}" ] || fs_opts+=( -b ${squahsfs_block} )
+
+    mksquashfs "${root_dir}" "${img}" "${fs_opts[@]}" >/dev/null
+}
+
+# vim: ft=sh
-- 
1.8.1.2

^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images
  2013-12-20 22:32 ` [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images Yann E. MORIN
@ 2013-12-21  8:49   ` Thomas Petazzoni
  2013-12-21 16:40     ` Peter Korsgaard
  2013-12-21 17:20     ` Yann E. MORIN
  0 siblings, 2 replies; 11+ messages in thread
From: Thomas Petazzoni @ 2013-12-21  8:49 UTC (permalink / raw)
  To: buildroot

Dear Yann E. MORIN,

On Fri, 20 Dec 2013 23:32:54 +0100, Yann E. MORIN wrote:

> +Requirements on host packages
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Depending on what types of partitioning and filesystems are used by your
> ++partition table layout description+, you may have to enable some
> +host-packages in the +Host utilities+ sub-menu.
> +
> +Since the content of a +partition table layout description+ is very
> +specific to a board and/or a project, there is no way for Buildroot to
> +automatically select those host packages, and thus it is your
> +responsibility to select the appropriate ones.
> +
> +For example, for an MBR partitioning, you will have to enable the +host
> +genpart+ package. For FAT filesystems, you will have to enable both of
> ++host dosfstools+ and +host mtools+.

I'll have other comments about the genimages stuff later on, but I got
an idea this night about this specific problem. I don't really like the
fact that the user has to manually enable in menuconfig the relevant
host tools, depending on his genimage .ini configuration file.

What about doing something like:

ROOTFS_CUSTOM_DEPENDENCIES = $(shell fs/custom/genimages --show-depends \
				'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'

The --show-depends option of genimages would, instead of doing the
normal genimages work, simply parse the .ini file, and return a space
separated list of dependencies needed by the ROOTFS_CUSTOM stuff. Not
only this would make the dependencies automatic for the user, but also
it would allow you to remove the following piece of code from the
custom.mk file:

+# All of the following filesystem generators, or partition managers, are
+# optional, but if they are selected, we may need them, so we need to
+# depend on them
+ifeq ($(BR2_PACKAGE_HOST_DOSFSTOOLS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-dosfstools
+endif
+ifeq ($(BR2_PACKAGE_HOST_E2FSPROGS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-e2fsprogs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENEXT2FS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-genext2fs
+endif
+ifeq ($(BR2_PACKAGE_HOST_GENPART),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-genpart
+endif
+ifeq ($(BR2_PACKAGE_HOST_MTOOLS),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-mtools
+endif
+ifeq ($(BR2_PACKAGE_HOST_PARTED),y)
+ROOTFS_CUSTOM_DEPENDENCIES += host-parted
+endif

What do you think?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images
  2013-12-21  8:49   ` Thomas Petazzoni
@ 2013-12-21 16:40     ` Peter Korsgaard
  2013-12-21 17:12       ` Yann E. MORIN
  2013-12-21 17:50       ` Thomas Petazzoni
  2013-12-21 17:20     ` Yann E. MORIN
  1 sibling, 2 replies; 11+ messages in thread
From: Peter Korsgaard @ 2013-12-21 16:40 UTC (permalink / raw)
  To: buildroot

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Dear Yann E. MORIN,
 > On Fri, 20 Dec 2013 23:32:54 +0100, Yann E. MORIN wrote:

 >> +Requirements on host packages
 >> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 >> +
 >> +Depending on what types of partitioning and filesystems are used by your
 >> ++partition table layout description+, you may have to enable some
 >> +host-packages in the +Host utilities+ sub-menu.
 >> +
 >> +Since the content of a +partition table layout description+ is very
 >> +specific to a board and/or a project, there is no way for Buildroot to
 >> +automatically select those host packages, and thus it is your
 >> +responsibility to select the appropriate ones.
 >> +
 >> +For example, for an MBR partitioning, you will have to enable the +host
 >> +genpart+ package. For FAT filesystems, you will have to enable both of
 >> ++host dosfstools+ and +host mtools+.

 > I'll have other comments about the genimages stuff later on, but I got
 > an idea this night about this specific problem. I don't really like the
 > fact that the user has to manually enable in menuconfig the relevant
 > host tools, depending on his genimage .ini configuration file.

 > What about doing something like:

 > ROOTFS_CUSTOM_DEPENDENCIES = $(shell fs/custom/genimages --show-depends \
 > 				'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'

 > The --show-depends option of genimages would, instead of doing the
 > normal genimages work, simply parse the .ini file, and return a space
 > separated list of dependencies needed by the ROOTFS_CUSTOM stuff. Not
 > only this would make the dependencies automatic for the user, but also
 > it would allow you to remove the following piece of code from the
 > custom.mk file:

Sounds good to me. They won't actumatically get selected in kconfig
though, but that might not be such a big deal (do they still even need
to be visible in kconfig?)

-- 
Bye, Peter Korsgaard

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images
  2013-12-21 16:40     ` Peter Korsgaard
@ 2013-12-21 17:12       ` Yann E. MORIN
  2013-12-21 17:50       ` Thomas Petazzoni
  1 sibling, 0 replies; 11+ messages in thread
From: Yann E. MORIN @ 2013-12-21 17:12 UTC (permalink / raw)
  To: buildroot

Thomas, Peter, All,

On 2013-12-21 17:40 +0100, Peter Korsgaard spake thusly:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> 
>  > Dear Yann E. MORIN,
>  > On Fri, 20 Dec 2013 23:32:54 +0100, Yann E. MORIN wrote:
> 
>  >> +Requirements on host packages
>  >> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>  >> +
>  >> +Depending on what types of partitioning and filesystems are used by your
>  >> ++partition table layout description+, you may have to enable some
>  >> +host-packages in the +Host utilities+ sub-menu.
>  >> +
>  >> +Since the content of a +partition table layout description+ is very
>  >> +specific to a board and/or a project, there is no way for Buildroot to
>  >> +automatically select those host packages, and thus it is your
>  >> +responsibility to select the appropriate ones.
>  >> +
>  >> +For example, for an MBR partitioning, you will have to enable the +host
>  >> +genpart+ package. For FAT filesystems, you will have to enable both of
>  >> ++host dosfstools+ and +host mtools+.
> 
>  > I'll have other comments about the genimages stuff later on, but I got
>  > an idea this night about this specific problem. I don't really like the
>  > fact that the user has to manually enable in menuconfig the relevant
>  > host tools, depending on his genimage .ini configuration file.
> 
>  > What about doing something like:
> 
>  > ROOTFS_CUSTOM_DEPENDENCIES = $(shell fs/custom/genimages --show-depends \
>  > 				'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
> 
>  > The --show-depends option of genimages would, instead of doing the
>  > normal genimages work, simply parse the .ini file, and return a space
>  > separated list of dependencies needed by the ROOTFS_CUSTOM stuff. Not
>  > only this would make the dependencies automatic for the user, but also
>  > it would allow you to remove the following piece of code from the
>  > custom.mk file:
> 
> Sounds good to me.

Yep, I've implemented that now, and it seems to work pretty well.

> They won't actumatically get selected in kconfig
> though, but that might not be such a big deal (do they still even need
> to be visible in kconfig?)

Yes, in case someone wants to generate filesystem images with a
post-build script, and not with genimages.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images
  2013-12-21  8:49   ` Thomas Petazzoni
  2013-12-21 16:40     ` Peter Korsgaard
@ 2013-12-21 17:20     ` Yann E. MORIN
  1 sibling, 0 replies; 11+ messages in thread
From: Yann E. MORIN @ 2013-12-21 17:20 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2013-12-21 09:49 +0100, Thomas Petazzoni spake thusly:
> On Fri, 20 Dec 2013 23:32:54 +0100, Yann E. MORIN wrote:
> 
> > +Requirements on host packages
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +Depending on what types of partitioning and filesystems are used by your
> > ++partition table layout description+, you may have to enable some
> > +host-packages in the +Host utilities+ sub-menu.
> > +
> > +Since the content of a +partition table layout description+ is very
> > +specific to a board and/or a project, there is no way for Buildroot to
> > +automatically select those host packages, and thus it is your
> > +responsibility to select the appropriate ones.
> > +
> > +For example, for an MBR partitioning, you will have to enable the +host
> > +genpart+ package. For FAT filesystems, you will have to enable both of
> > ++host dosfstools+ and +host mtools+.
> 
> I'll have other comments about the genimages stuff later on,

Hey! You can't let us all waiting like that! Speak up, now! ;-)

> but I got an idea this night

I don't know what you usually do at night, but I, for one, sleep. ;-)

> about this specific problem. I don't really like the
> fact that the user has to manually enable in menuconfig the relevant
> host tools, depending on his genimage .ini configuration file.
> 
> What about doing something like:
> 
> ROOTFS_CUSTOM_DEPENDENCIES = $(shell fs/custom/genimages --show-depends \
> 				'$(call qstrip,$(BR2_TARGET_ROOTFS_CUSTOM_PARTITION_TABLE))'
[--SNIP--]
> What do you think?

I was initially thinking about something like it, but found it a bit
dubious. But since you suggested it (and now Peter concurred), I have
now (re-)implemented it, and it works nicely.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images
  2013-12-21 16:40     ` Peter Korsgaard
  2013-12-21 17:12       ` Yann E. MORIN
@ 2013-12-21 17:50       ` Thomas Petazzoni
  1 sibling, 0 replies; 11+ messages in thread
From: Thomas Petazzoni @ 2013-12-21 17:50 UTC (permalink / raw)
  To: buildroot

Dear Peter Korsgaard,

On Sat, 21 Dec 2013 17:40:33 +0100, Peter Korsgaard wrote:

>  > The --show-depends option of genimages would, instead of doing the
>  > normal genimages work, simply parse the .ini file, and return a
>  > space separated list of dependencies needed by the ROOTFS_CUSTOM
>  > stuff. Not only this would make the dependencies automatic for the
>  > user, but also it would allow you to remove the following piece of
>  > code from the custom.mk file:
> 
> Sounds good to me. They won't actumatically get selected in kconfig
> though, but that might not be such a big deal (do they still even need
> to be visible in kconfig?)

I believe we have existing occurrences of host packages being listed in
some <pkg>_DEPENDENCIES, but not necessarily selected at the Kconfig
level. While we have enforced this not to happen for target packages,
we've never enforced that for host packages.

The rationale is that host packages should normally be completely
transparent to the user, and therefore not visible in Kconfig. The host
package sub-menu is here only to make it easier for the user to enable
*some* of the host packages that are actually relevant for the user
(and not merely build dependencies of other packages).

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2013-12-21 17:50 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-20 22:32 [Buildroot] [pull request v3] Pull request for branch yem/genimages Yann E. MORIN
2013-12-20 22:32 ` [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images Yann E. MORIN
2013-12-21  8:49   ` Thomas Petazzoni
2013-12-21 16:40     ` Peter Korsgaard
2013-12-21 17:12       ` Yann E. MORIN
2013-12-21 17:50       ` Thomas Petazzoni
2013-12-21 17:20     ` Yann E. MORIN
2013-12-20 22:32 ` [Buildroot] [PATCH 2/4] board/raspberrypi: provide partition description for the new genimages Yann E. MORIN
2013-12-20 22:32 ` [Buildroot] [PATCH 3/4] package/squashfs: add selection for the host variant Yann E. MORIN
2013-12-20 22:32 ` [Buildroot] [PATCH 4/4] fs/custom: add support for squashfs Yann E. MORIN
  -- strict thread matches above, loose matches on Subject: below --
2013-12-16 22:26 [Buildroot] [RFC v2] Introduce the genimages infrastructure Yann E. MORIN
2013-12-16 22:26 ` [Buildroot] [PATCH 1/4] fs/custom: generate complete, partition-based device images Yann E. MORIN

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox