[PATCH 0/2] support numeric owner/group tar image format

[PATCH 0/2] support numeric owner/group tar image format

[Peter A. Bigot]

A classic approach to building SD card images with varying rootfs sizes
is to create the filesystem and unpack a tar rootfs image onto it.  When
done outside pseudo the encoded user and group names from the tar file
are converted to identifiers using host files, which often result in
incorrect assignment for common names like messagebus and ntp.

A solution is to create the rootfs archive using the numeric values
obtained from the target /etc files under pseudo, rather than the
corresponding names.  This series provides this function as a new image
file format "nug.tar".

Peter A. Bigot (2):
  image_types.bbclass: whitespace and reorder
  image_types.bbclass: add tar --numeric-owner support

 meta/classes/image_types.bbclass | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

-- 
1.8.5.5

[PATCH 1/2] image_types.bbclass: whitespace and reorder

[Peter A. Bigot]

Use a multi-line value to set IMAGE_TYPES and put all the compressed
versions of a format on one line.  Other than moving ext2.lzma this is
just whitespace.

Signed-off-by: Peter A. Bigot <pab@pabigot.com>
---
 meta/classes/image_types.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index d298897..42d7231 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -128,7 +128,21 @@ IMAGE_DEPENDS_ubi = "mtd-utils-native"
 IMAGE_DEPENDS_ubifs = "mtd-utils-native"
 
 # This variable is available to request which values are suitable for IMAGE_FSTYPES
-IMAGE_TYPES = "jffs2 jffs2.sum cramfs ext2 ext2.gz ext2.bz2 ext3 ext3.gz ext2.lzma btrfs iso hddimg squashfs squashfs-xz squashfs-lzo ubi ubifs tar tar.gz tar.bz2 tar.xz tar.lz4 cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 vmdk elf"
+IMAGE_TYPES = " \
+    jffs2 jffs2.sum \
+    cramfs \
+    ext2 ext2.gz ext2.bz2 ext2.lzma \
+    ext3 ext3.gz \
+    btrfs \
+    iso \
+    hddimg \
+    squashfs squashfs-xz squashfs-lzo \
+    ubi ubifs \
+    tar tar.gz tar.bz2 tar.xz tar.lz4 \
+    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
+    vmdk \
+    elf \
+"
 
 COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
 COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}.rootfs.${type}"
-- 
1.8.5.5

[PATCH 2/2] image_types.bbclass: add tar --numeric-owner support

[Peter A. Bigot]

tar format normally stores user and group as names, which is the right
thing to use when target passwd and group files are available.  When
unpacking a rootfs archive onto a mounted SD card partition on a build
host outside the pseudo environment the host passwd/group files will be
used for name-to-id mapping, which results in mis-assigned identifiers
(often for important ids like messagebus).

Using IMAGE_FSTYPES += "nug.tar" creates rootfs.nug.tar files where the
owner and group are specified numerically, using the target IDs obtained
when the archive is built under pseudo.

Signed-off-by: Peter A. Bigot <pab@pabigot.com>
---
 meta/classes/image_types.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 42d7231..50657ab 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -65,6 +65,7 @@ IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAM
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
 IMAGE_CMD_tar = "tar -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar -C ${IMAGE_ROOTFS} ."
+IMAGE_CMD_nug.tar = "tar --numeric-owner -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.nug.tar -C ${IMAGE_ROOTFS} ."
 
 IMAGE_CMD_cpio () {
 	(cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc >${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.cpio)
@@ -139,6 +140,7 @@ IMAGE_TYPES = " \
     squashfs squashfs-xz squashfs-lzo \
     ubi ubifs \
     tar tar.gz tar.bz2 tar.xz tar.lz4 \
+    nug.tar nug.tar.gz nug.tar.bz2 nug.tar.xz nug.tar.lz4 \
     cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
     vmdk \
     elf \
-- 
1.8.5.5

Re: [PATCH 2/2] image_types.bbclass: add tar --numeric-owner support

[Richard Purdie]

On Sun, 2014-11-02 at 04:48 -0600, Peter A. Bigot wrote:
> tar format normally stores user and group as names, which is the right
> thing to use when target passwd and group files are available.  When
> unpacking a rootfs archive onto a mounted SD card partition on a build
> host outside the pseudo environment the host passwd/group files will be
> used for name-to-id mapping, which results in mis-assigned identifiers
> (often for important ids like messagebus).
> 
> Using IMAGE_FSTYPES += "nug.tar" creates rootfs.nug.tar files where the
> owner and group are specified numerically, using the target IDs obtained
> when the archive is built under pseudo.
> 
> Signed-off-by: Peter A. Bigot <pab@pabigot.com>
> ---
>  meta/classes/image_types.bbclass | 2 ++
>  1 file changed, 2 insertions(+)

Shouldn't we always do this?

I'm trying to figure out when this would be a bad idea...

Cheers,

Richard

Re: [PATCH 2/2] image_types.bbclass: add tar --numeric-owner support

[Peter A. Bigot]

On 11/02/2014 09:01 AM, Richard Purdie wrote:
> On Sun, 2014-11-02 at 04:48 -0600, Peter A. Bigot wrote:
>> tar format normally stores user and group as names, which is the right
>> thing to use when target passwd and group files are available.  When
>> unpacking a rootfs archive onto a mounted SD card partition on a build
>> host outside the pseudo environment the host passwd/group files will be
>> used for name-to-id mapping, which results in mis-assigned identifiers
>> (often for important ids like messagebus).
>>
>> Using IMAGE_FSTYPES += "nug.tar" creates rootfs.nug.tar files where the
>> owner and group are specified numerically, using the target IDs obtained
>> when the archive is built under pseudo.
>>
>> Signed-off-by: Peter A. Bigot <pab@pabigot.com>
>> ---
>>   meta/classes/image_types.bbclass | 2 ++
>>   1 file changed, 2 insertions(+)
> Shouldn't we always do this?
>
> I'm trying to figure out when this would be a bad idea...

I read 
http://www.mail-archive.com/openembedded-core%40lists.openembedded.org/msg45147.html 
as suggesting you thought otherwise.

But no, I think using names is generally appropriate.  It's what the 
underlying packages use (well, RPM at least; I didn't check others).  
It's nice in that it does allow you to see the owner and group from tar 
-tav without having to unpack /etc to get the uid/gid mappings.  If you 
unpack it on a development machine to see what's in it, chances are the 
files will end up owned by the right users. And it works just fine in 
the standard Yocto environment that uses pseudo to map names, as 
(perhaps) wic using it to create a new custom rootfs might do.

Pretty much the only situation where it doesn't work is unpacking it as 
root not using pseudo in a context you really want the resulting uid/gid 
to be what they'll be on the target.  That's a pretty important use 
case, but not IMO enough to change the default from what it's "always 
been", especially as it'll go away once wic matures a little.

Peter

Re: [PATCH 0/2] support numeric owner/group tar image format

[Mark Hatle]

On 11/2/14, 4:48 AM, Peter A. Bigot wrote:
> A classic approach to building SD card images with varying rootfs sizes
> is to create the filesystem and unpack a tar rootfs image onto it.  When
> done outside pseudo the encoded user and group names from the tar file
> are converted to identifiers using host files, which often result in
> incorrect assignment for common names like messagebus and ntp.
>
> A solution is to create the rootfs archive using the numeric values
> obtained from the target /etc files under pseudo, rather than the
> corresponding names.  This series provides this function as a new image
> file format "nug.tar".
>
> Peter A. Bigot (2):
>    image_types.bbclass: whitespace and reorder
>    image_types.bbclass: add tar --numeric-owner support
>
>   meta/classes/image_types.bbclass | 18 +++++++++++++++++-
>   1 file changed, 17 insertions(+), 1 deletion(-)
>

I think this is incorrect.

The current tar archives contain both the uname/gname -and- uid/gid.  When you 
extract, if you pass --numeric-owner it will switch from name to number.  (At 
least it has whenever I've done it.)  The default though is to use the uname/gname.

The issue with switching to uid/gid is that you can end up changing the default 
and you may enough end up dumping the name version.  This can cause security 
issues if you do partial extracts and there was some type of a mismatch between 
the original system and the one being extracted.  (Think field 
upgrade/debug/update cycle.)

--Mark

Re: [PATCH 0/2] support numeric owner/group tar image format

[Peter A. Bigot]

On 11/03/2014 08:55 AM, Mark Hatle wrote:
> On 11/2/14, 4:48 AM, Peter A. Bigot wrote:
>> A classic approach to building SD card images with varying rootfs sizes
>> is to create the filesystem and unpack a tar rootfs image onto it.  When
>> done outside pseudo the encoded user and group names from the tar file
>> are converted to identifiers using host files, which often result in
>> incorrect assignment for common names like messagebus and ntp.
>>
>> A solution is to create the rootfs archive using the numeric values
>> obtained from the target /etc files under pseudo, rather than the
>> corresponding names.  This series provides this function as a new image
>> file format "nug.tar".
>>
>> Peter A. Bigot (2):
>>    image_types.bbclass: whitespace and reorder
>>    image_types.bbclass: add tar --numeric-owner support
>>
>>   meta/classes/image_types.bbclass | 18 +++++++++++++++++-
>>   1 file changed, 17 insertions(+), 1 deletion(-)
>>
>
> I think this is incorrect.
>
> The current tar archives contain both the uname/gname -and- uid/gid.  
> When you extract, if you pass --numeric-owner it will switch from name 
> to number.  (At least it has whenever I've done it.)  The default 
> though is to use the uname/gname.

You're correct; both are stored per: 
http://www.gnu.org/software/tar/manual/html_node/Standard.html

This should probably be publicized more widely, since when the issue 
came up recently I wasn't the only one who discovered the hand-generated 
images had the wrong uid/gid values for non-root files.  Unnecessarily, 
as it turns out: adding --numeric-owner to the unpack does solve the 
problem.

Thanks.  I'll archive both patches in patchwork (the first is still 
valid, but is now unmotivated).

Peter