[PATCH 0/4][edison] Fix bootimg bug 2138 for 1.1.2

[PATCH 0/4][edison] Fix bootimg bug 2138 for 1.1.2

[Darren Hart]

Backport the following patches to edison. The backport involves massaging
this minimal set of patches to apply to bootimg.bbclass without pulling
in the refactoring or EFI support added in master about the same time.

These patches have been tested by building atom-pc core-image-minimal with
edison, a dosfsck and mount test on resulting image, and booting on the n450.

The following changes since commit 0fbd6a161576b2cafa8583adde0ffb15347c884a:

  poky.conf: Bumping SKD_VERSION (2012-03-14 15:49:33 -0700)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib dvhart/edison/bug/2138
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dvhart/edison/bug/2138

Darren Hart (4):
  bootimg: Use mcopy to construct the hddimg
  bootimg: Account for FAT filesystem overhead in image size
  bootimg: Fix a math thinko in the block count calculation
  bootimg: Do not force FAT32 on all images, it violates the FAT
    specification

 meta/classes/bootimg.bbclass |   53 ++++++++++++++++++++++++++++++++++++-----
 1 files changed, 46 insertions(+), 7 deletions(-)

-- 
1.7.5.4

[PATCH 1/4] bootimg: Use mcopy to construct the hddimg

[Darren Hart]

Fixes [YOCTO 2138]

The initial directory support (-d) added to mkdosfs has proven to be incomplete
and non-compliant with FAT. Rather than continue to maintain this feature and
work around the various issues, we can use mcopy to construct the image.

bootimg.bbclass already depends on mtools-native (although it may not have
needed to previously). No new dependencies are introduced. The image created
passes dosfsck cleanly. Remove the call to dosfsck.

mcopy reported an error with the image we were creating:
Total number of sectors (107574) not a multiple of sectors per track (32)!

Add some logic to ensure the total sector count is an integral number of sectors
per track, including forcing the logical sector size to 512 in the mkdosfs
command.

The du -bks arguments are contradictory, -b is equivalent to "--apparent-size
--block-size=1" and -k is --block-size=1K. If reordered, -kbs will report the
disk usage in bytes insteadk of 1k blocks. Eliminate the ambiguity by using:
du --apparent-size -ks

(From OE-Core rev: 92d2ea1a306354c6565a1b05b51b5719e481840f)

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
CC: Nitin A. Kamble <nitin.a.kamble@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Backported to poky edison by Darren Hart.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
---
 meta/classes/bootimg.bbclass |   22 +++++++++++++++-------
 1 files changed, 15 insertions(+), 7 deletions(-)

diff --git a/meta/classes/bootimg.bbclass b/meta/classes/bootimg.bbclass
index 49ee85e..bcae2ae 100644
--- a/meta/classes/bootimg.bbclass
+++ b/meta/classes/bootimg.bbclass
@@ -62,13 +62,21 @@ build_boot_bin() {
 
 	install -m 444 ${STAGING_LIBDIR}/syslinux/ldlinux.sys ${HDDDIR}/ldlinux.sys
 
-	# Do a little math, bash style
-	#BLOCKS=`du -s ${HDDDIR} | cut -f 1`
-	BLOCKS=`du -bks ${HDDDIR} | cut -f 1`
-	SIZE=`expr $BLOCKS + ${BOOTIMG_EXTRA_SPACE}`	
-
-	mkdosfs -n ${BOOTIMG_VOLUME_ID} -d ${HDDDIR} \
-	-C ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg $SIZE 
+	# Determine the 1024 byte block count for the final image.
+	BLOCKS=`du --apparent-size -ks ${HDDDIR} | cut -f 1`
+	SIZE=`expr $BLOCKS + ${BOOTIMG_EXTRA_SPACE}`
+
+	# Ensure total sectors is an integral number of sectors per
+	# track or mcopy will complain. Sectors are 512 bytes, and and
+	# we generate images with 32 sectors per track. This calculation
+	# is done in blocks, which are twice the size of sectors, thus
+	# the 16 instead of 32.
+	SIZE=$(expr $SIZE + $(expr 16 - $(expr $SIZE % 16)))
+
+	IMG=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
+	mkdosfs -n ${BOOTIMG_VOLUME_ID} -S 512 -C ${IMG} ${SIZE}
+	# Copy HDDDIR recursively into the image file directly
+	mcopy -i ${IMG} -s ${HDDDIR}/* ::/
 
 	syslinux ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
 	chmod 644 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
-- 
1.7.5.4

[PATCH 2/4] bootimg: Account for FAT filesystem overhead in image size

[Darren Hart]

Fixes [YOCTO #1852]

The bootimg class wasn't accounting for non-trivial amount of space
required by the directory entries and FATs for the FAT filesystem.

This patch attempts to make an accurate prediction of FAT overhead and
adjusts the image size accordingly. It assumes no more than 16 directory
entries per directory (which fit in a single sector). It also assumes
8.3 filenames. With the ceiling functions rounding up to full sectors
and tracks, these assumptions seem reasonable.

In order to ensure the calculations are accurate, this patch forces the
FAT size to 32, rather than allowing mkdosfs to automatically select 12,
16, or 32 depending on the image being built.

Tested by setting BOOTIMG_EXTRA_SPACE=0 and building core-image-minimal
and core-image-sato for fri2-noemgd from meta-intel.

(From OE-Core rev: 68aa18609c10a3ae2f738930c933fa2a95ce8959)

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
CC: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Backported to edison by Darren Hart.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
---
 meta/classes/bootimg.bbclass |   45 +++++++++++++++++++++++++++++++++--------
 1 files changed, 36 insertions(+), 9 deletions(-)

diff --git a/meta/classes/bootimg.bbclass b/meta/classes/bootimg.bbclass
index bcae2ae..c2d43c8 100644
--- a/meta/classes/bootimg.bbclass
+++ b/meta/classes/bootimg.bbclass
@@ -62,19 +62,46 @@ build_boot_bin() {
 
 	install -m 444 ${STAGING_LIBDIR}/syslinux/ldlinux.sys ${HDDDIR}/ldlinux.sys
 
-	# Determine the 1024 byte block count for the final image.
-	BLOCKS=`du --apparent-size -ks ${HDDDIR} | cut -f 1`
-	SIZE=`expr $BLOCKS + ${BOOTIMG_EXTRA_SPACE}`
+	# Calculate the size required for the final image including the
+	# data and filesystem overhead.
+	# Sectors: 512 bytes
+	#  Blocks: 1024 bytes
+
+	# Determine the sector count just for the data
+	SECTORS=$(expr $(du --apparent-size -ks ${HDDDIR} | cut -f 1) \* 2)
+
+	# Account for the filesystem overhead. This includes directory
+	# entries in the clusters as well as the FAT itself.
+	# Assumptions:
+	#   < 16 entries per directory
+	#   8.3 filenames only
+
+	# 32 bytes per dir entry
+	DIR_BYTES=$(expr $(find ${HDDDIR} | tail -n +2 | wc -l) \* 32)
+	# 32 bytes for every end-of-directory dir entry
+	DIR_BYTES=$(expr $DIR_BYTES + $(expr $(find ${HDDDIR} -type d | tail -n +2 | wc -l) \* 32))
+	# 4 bytes per FAT entry per sector of data
+	FAT_BYTES=$(expr $SECTORS \* 4)
+	# 4 bytes per FAT entry per end-of-cluster list
+	FAT_BYTES=$(expr $FAT_BYTES + $(expr $(find ${HDDDIR} -type d | tail -n +2 | wc -l) \* 4))
+
+	# Use a ceiling function to determine FS overhead in sectors
+	DIR_SECTORS=$(expr $(expr $DIR_BYTES + 511) / 512)
+	# There are two FATs on the image
+	FAT_SECTORS=$(expr $(expr $(expr $FAT_BYTES + 511) / 512) \* 2)
+	SECTORS=$(expr $SECTORS + $(expr $DIR_SECTORS + $FAT_SECTORS))
+
+	# Determine the final size in blocks accounting for some padding
+	BLOCKS=$(expr $(expr $SECTORS \* 2) + ${BOOTIMG_EXTRA_SPACE})
 
 	# Ensure total sectors is an integral number of sectors per
-	# track or mcopy will complain. Sectors are 512 bytes, and and
-	# we generate images with 32 sectors per track. This calculation
-	# is done in blocks, which are twice the size of sectors, thus
-	# the 16 instead of 32.
-	SIZE=$(expr $SIZE + $(expr 16 - $(expr $SIZE % 16)))
+	# track or mcopy will complain. Sectors are 512 bytes, and we
+	# generate images with 32 sectors per track. This calculation is
+	# done in blocks, thus the mod by 16 instead of 32.
+	BLOCKS=$(expr $BLOCKS + $(expr 16 - $(expr $BLOCKS % 16)))
 
 	IMG=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
-	mkdosfs -n ${BOOTIMG_VOLUME_ID} -S 512 -C ${IMG} ${SIZE}
+	mkdosfs -F 32 -n ${BOOTIMG_VOLUME_ID} -S 512 -C ${IMG} ${BLOCKS}
 	# Copy HDDDIR recursively into the image file directly
 	mcopy -i ${IMG} -s ${HDDDIR}/* ::/
 
-- 
1.7.5.4

[PATCH 3/4] bootimg: Fix a math thinko in the block count calculation

[Darren Hart]

Fixes [YOCTO #1852] ... again.

The conversion from sectors to blocks was multiplying by 2 instead
of dividing by 2. Blocks are 1024 bytes, sectors are 512 bytes. The
result was images being much larger than intended.

Reported-by: Tom Zanussi <tom.zanussi@intel.com>
(From OE-Core rev: b35384fa3ca96b31c63d764322215abced2066e4)

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
CC: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Backported to edison by Darren Hart.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
---
 meta/classes/bootimg.bbclass |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/meta/classes/bootimg.bbclass b/meta/classes/bootimg.bbclass
index c2d43c8..489819b 100644
--- a/meta/classes/bootimg.bbclass
+++ b/meta/classes/bootimg.bbclass
@@ -92,7 +92,8 @@ build_boot_bin() {
 	SECTORS=$(expr $SECTORS + $(expr $DIR_SECTORS + $FAT_SECTORS))
 
 	# Determine the final size in blocks accounting for some padding
-	BLOCKS=$(expr $(expr $SECTORS \* 2) + ${BOOTIMG_EXTRA_SPACE})
+	BLOCKS=$(expr $(expr $SECTORS / 2) + ${BOOTIMG_EXTRA_SPACE})
+
 
 	# Ensure total sectors is an integral number of sectors per
 	# track or mcopy will complain. Sectors are 512 bytes, and we
-- 
1.7.5.4

[PATCH 4/4] bootimg: Do not force FAT32 on all images, it violates the FAT specification

[Darren Hart]

Fixes [YOCTO #1940]

do_bootimg was performing the FAT overhead calculations assuming FAT32 and then
forcing the use of FAT32 with "-F 32" to mkdosfs. The FAT specification is clear
on cluster count being the determining factor for FAT size (even if the fs
string is set to FAT32, go figure). Syslinux follows this spec, and rightly so,
resulting in a failure on core-image-minimal:

	syslinux: zero FAT sectors (FAT12/16)

Drop the "-F 32" from mkdosfs to allow it to select the appropriate FAT size
based on cluster count. Leave the FAT overhead calculation in FAT32. This will
result in a little extra padding for really small images, but not enough extra
to justify recalculating for FAT12 and FAT16.

Tested with a core-image-minimal build for atom-pc. do_bootimg completed
successfully, and the resulting image was FAT16.

(From OE-Core rev: 634137704dd1a205e377a1131ef708f1c981f6b2)

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Backported to edison by Darren Hart.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
---
 meta/classes/bootimg.bbclass |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/meta/classes/bootimg.bbclass b/meta/classes/bootimg.bbclass
index 489819b..09f5b67 100644
--- a/meta/classes/bootimg.bbclass
+++ b/meta/classes/bootimg.bbclass
@@ -73,6 +73,9 @@ build_boot_bin() {
 	# Account for the filesystem overhead. This includes directory
 	# entries in the clusters as well as the FAT itself.
 	# Assumptions:
+	#   FAT32 (12 or 16 may be selected by mkdosfs, but the extra
+	#   padding will be minimal on those smaller images and not
+	#   worth the logic here to caclulate the smaller FAT sizes)
 	#   < 16 entries per directory
 	#   8.3 filenames only
 
@@ -102,7 +105,7 @@ build_boot_bin() {
 	BLOCKS=$(expr $BLOCKS + $(expr 16 - $(expr $BLOCKS % 16)))
 
 	IMG=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hddimg
-	mkdosfs -F 32 -n ${BOOTIMG_VOLUME_ID} -S 512 -C ${IMG} ${BLOCKS}
+	mkdosfs -n ${BOOTIMG_VOLUME_ID} -S 512 -C ${IMG} ${BLOCKS}
 	# Copy HDDDIR recursively into the image file directly
 	mcopy -i ${IMG} -s ${HDDDIR}/* ::/
 
-- 
1.7.5.4

Re: [PATCH 0/4][edison] Fix bootimg bug 2138 for 1.1.2

[Joshua Lock]

On 05/04/12 14:09, Darren Hart wrote:
> Backport the following patches to edison. The backport involves massaging
> this minimal set of patches to apply to bootimg.bbclass without pulling
> in the refactoring or EFI support added in master about the same time.
>
> These patches have been tested by building atom-pc core-image-minimal with
> edison, a dosfsck and mount test on resulting image, and booting on the n450.
>
> The following changes since commit 0fbd6a161576b2cafa8583adde0ffb15347c884a:
>
>    poky.conf: Bumping SKD_VERSION (2012-03-14 15:49:33 -0700)
>
> are available in the git repository at:
>    git://git.pokylinux.org/poky-contrib dvhart/edison/bug/2138
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dvhart/edison/bug/2138
>
> Darren Hart (4):
>    bootimg: Use mcopy to construct the hddimg
>    bootimg: Account for FAT filesystem overhead in image size
>    bootimg: Fix a math thinko in the block count calculation
>    bootimg: Do not force FAT32 on all images, it violates the FAT
>      specification
>
>   meta/classes/bootimg.bbclass |   53 ++++++++++++++++++++++++++++++++++++-----
>   1 files changed, 46 insertions(+), 7 deletions(-)
>

Thanks! I've merged these to my Edison test branch and (assuming testing 
goes well) will submit a consolidated pull shortly.

Cheers,
Joshua
-- 
Joshua '贾詡' Lock
         Yocto Project "Johannes factotum"
         Intel Open Source Technology Centre