[PATCH 0/3] preserve xattrs in images

[PATCH 0/3] preserve xattrs in images

[Patrick Ohly]

Both Smack and IMA/EVM rely on xattrs in the rootfs. This works for
.ext3/.ext4 images, but not for .jffs2 and .tar.bz2. These changes
allow optionally building also such images with xattrs without
changing the default (which still is to ignore xattrs in .jffs2 and
.tar.bz2).

The default does not get changed because supporting xattrs causes a
certain overhead (need to build GNU tar, additional system calls when
creating the images).

See https://github.com/01org/meta-intel-iot-security/pull/34 for code using
these changes.

The following changes since commit 5094354a2811825e6d60963f03959daa349cab23:

  bind: upgrade to 9.10.2-p3 (2015-08-09 15:14:32 -0700)

are available in the git repository at:

  git://github.com/pohly/openembedded-core xattr
  https://github.com/pohly/openembedded-core/tree/xattr

Patrick Ohly (3):
  tar-replacement-native: avoid race condition with host tar
  image_types.bbclass: allow replacing tar command
  mtd-utils: keep xattr support enabled

 meta/classes/image_types.bbclass                         | 13 ++++++++++++-
 meta/recipes-devtools/mtd/mtd-utils_git.bb               |  2 +-
 meta/recipes-extended/tar/tar-replacement-native_1.28.bb | 14 +++++++++++++-
 3 files changed, 26 insertions(+), 3 deletions(-)

-- 
2.1.4

[PATCH 1/3] tar-replacement-native: avoid race condition with host tar

[Patrick Ohly]

Installing "tar" into the sysroot leads to race conditions
(tasks which do not depend on tar-replacement-native may already
call tar while it's installation is incomplete). Avoid those
by installing only the tar binary under the name tar-native.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/recipes-extended/tar/tar-replacement-native_1.28.bb | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-extended/tar/tar-replacement-native_1.28.bb b/meta/recipes-extended/tar/tar-replacement-native_1.28.bb
index 071ede7..d69122f 100644
--- a/meta/recipes-extended/tar/tar-replacement-native_1.28.bb
+++ b/meta/recipes-extended/tar/tar-replacement-native_1.28.bb
@@ -3,4 +3,16 @@ require tar_${PV}.bb
 inherit native
 
 BPN = "tar"
-EXTRAINSTALL = ""
+
+# Installing "tar" into the sysroot leads to race conditions
+# (tasks which do not depend on tar-replacement-native may already
+# call tar while it's installation is incomplete). Avoid those
+# by installing only the tar binary under the name tar-native.
+EXTRAINSTALL = "do_install_extra_native"
+do_install_extra_native () {
+    remove=$(ls -d ${D}/*)
+    mv ${D}${bindir}/tar ${D}/tar-native
+    rm -r $remove
+    install -d ${D}${bindir}
+    mv ${D}/tar-native ${D}${bindir}
+}
-- 
2.1.4

[PATCH 2/3] image_types.bbclass: allow replacing tar command

[Patrick Ohly]

Usually, the host's tar command is sufficient. However, special cases
like archiving xattrs depend on a modern GNU tar version. The new
IMAGE_CMD_TAR makes that possible, with xattrs given as example.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/image_types.bbclass | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index cc789fc..752fd52 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -71,7 +71,18 @@ IMAGE_CMD_btrfs () {
 IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs ${EXTRA_IMAGECMD} -noappend"
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
-IMAGE_CMD_tar = "tar -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar -C ${IMAGE_ROOTFS} ."
+
+# By default, tar from the host is used, which can be quite old. If
+# you need special parameters (like --xattrs) which are only supported
+# by GNU tar upstream >= 1.27, then override that default:
+# IMAGE_CMD_TAR = "tar-native --xattrs --xattrs-include=*"
+# IMAGE_DEPENDS_tar_append = " tar-replacement-native"
+#
+# The GNU documentation does not specify whether --xattrs-include is necessary.
+# In practice, it turned out to be not needed when creating archives and
+# required when extracting, but it seems prudent to use it in both cases.
+IMAGE_CMD_TAR ?= "tar"
+IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar -C ${IMAGE_ROOTFS} ."
 
 do_rootfs[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
-- 
2.1.4

[PATCH 3/3] mtd-utils: keep xattr support enabled

[Patrick Ohly]

xattrs may be needed by some distros. Support that by compiling in the
necessary code, even if it is not used by default. Then .jffs2 images
including xattrs can be created with:

   EXTRA_IMAGECMD_jffs2_append = " --with-xattr"

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/recipes-devtools/mtd/mtd-utils_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/mtd/mtd-utils_git.bb b/meta/recipes-devtools/mtd/mtd-utils_git.bb
index 7010cac..8d4892a 100644
--- a/meta/recipes-devtools/mtd/mtd-utils_git.bb
+++ b/meta/recipes-devtools/mtd/mtd-utils_git.bb
@@ -19,7 +19,7 @@ SRC_URI = "git://git.infradead.org/mtd-utils.git \
 
 S = "${WORKDIR}/git/"
 
-EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
+EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include' 'BUILDDIR=${S}'"
 
 do_install () {
 	oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir}
-- 
2.1.4

Re: [PATCH 0/3] preserve xattrs in images

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

Hi Patrick,

On 11 August 2015 at 09:44, Patrick Ohly <patrick.ohly@intel.com> wrote:

> The default does not get changed because supporting xattrs causes a
> certain overhead (need to build GNU tar, additional system calls when
> creating the images).
>

Two questions:
1) Do enough host distributions not enable xattrs in tar that we need to
depend on tar-replacement-native?
2) Have you timed the overhead of enabling xattr archiving on an image that
doesn't use xattrs?  (subtext: does this need to be an option).

Ross

[-- Attachment #2: Type: text/html, Size: 1041 bytes --]

Re: [PATCH 3/3] mtd-utils: keep xattr support enabled

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

On 11 August 2015 at 09:45, Patrick Ohly <patrick.ohly@intel.com> wrote:

> -EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}'
> 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
> +EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}'
> 'CFLAGS=${CFLAGS} -I${S}/include' 'BUILDDIR=${S}'"
>

There's a "xattr" DISTRO_FEATURE that should be respected here for target
builds (and explicitly enabled for native I guess).

Does enabling xattr mean adding build dependencies?

Ross

[-- Attachment #2: Type: text/html, Size: 1038 bytes --]

Re: [PATCH 0/3] preserve xattrs in images

[Patrick Ohly]

On Tue, 2015-08-11 at 15:29 +0100, Burton, Ross wrote:
> Hi Patrick,
> 
> On 11 August 2015 at 09:44, Patrick Ohly <patrick.ohly@intel.com>
> wrote:
>         The default does not get changed because supporting xattrs
>         causes a
>         certain overhead (need to build GNU tar, additional system
>         calls when
>         creating the images).
> 
> 
> Two questions:
> 1) Do enough host distributions not enable xattrs in tar that we need
> to depend on tar-replacement-native?

Yes, some of the currently supported host distributions (like Debian 7)
have a GNU tar which is too old.

> 2) Have you timed the overhead of enabling xattr archiving on an image
> that doesn't use xattrs?  (subtext: does this need to be an option).

I hadn't, because my concern was that there might be other build
configurations (for example, with less RAM for the page cache) where the
effect will be more pronounced than on my own system.

Anyway, some benchmark data: with a rootfs of around 400MB and 14000
files and directories and xattrs on all regular files, "tar
-cf /dev/null" takes around 1s wall clock time when the entire rootfs
tree is already in the page cache. With archiving xattrs, it is around 3
seconds. If there no xattrs on the file, that last value drops a bit to
something between 2 and 3.

So the relative runtime overhead is noticeable, but it is negligible in
absolute terms.

More important is that GNU tar needs to be built. That takes about a
minute of wall clock time here (most of it in configure, actual
compilation is parallel and fast thanks for many cores).

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 3/3] mtd-utils: keep xattr support enabled

[Patrick Ohly]

On Tue, 2015-08-11 at 15:33 +0100, Burton, Ross wrote:
> 
> On 11 August 2015 at 09:45, Patrick Ohly <patrick.ohly@intel.com>
> wrote:
>         -EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}'
>         'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=
>         ${S}'"
>         +EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}'
>         'CFLAGS=${CFLAGS} -I${S}/include' 'BUILDDIR=${S}'"
> 
> 
> There's a "xattr" DISTRO_FEATURE that should be respected here for
> target builds (and explicitly enabled for native I guess).

I can add that check, if you want.

> Does enabling xattr mean adding build dependencies?

No, the conditional code uses lgetxattr and llistxattr, which are in
glibc.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 0/3] preserve xattrs in images

[Mark Hatle]

Is there something here that enables the tar-replacement-native?  Or is the user
just expected to know they need it?

If you look at meta/classes/sanity.bbclass, there is already a tar check for a
specific old version that can't handle overwriting symlinks properly.  It should
be possible to check for the distro feature of xattr and that the host tar has
this.  It would be good to prevent the user from having a problem early (sanity)
rather then at filesystem generation time.

--Mark

On 8/11/15 3:44 AM, Patrick Ohly wrote:
> Both Smack and IMA/EVM rely on xattrs in the rootfs. This works for
> .ext3/.ext4 images, but not for .jffs2 and .tar.bz2. These changes
> allow optionally building also such images with xattrs without
> changing the default (which still is to ignore xattrs in .jffs2 and
> .tar.bz2).
> 
> The default does not get changed because supporting xattrs causes a
> certain overhead (need to build GNU tar, additional system calls when
> creating the images).
> 
> See https://github.com/01org/meta-intel-iot-security/pull/34 for code using
> these changes.
> 
> The following changes since commit 5094354a2811825e6d60963f03959daa349cab23:
> 
>   bind: upgrade to 9.10.2-p3 (2015-08-09 15:14:32 -0700)
> 
> are available in the git repository at:
> 
>   git://github.com/pohly/openembedded-core xattr
>   https://github.com/pohly/openembedded-core/tree/xattr
> 
> Patrick Ohly (3):
>   tar-replacement-native: avoid race condition with host tar
>   image_types.bbclass: allow replacing tar command
>   mtd-utils: keep xattr support enabled
> 
>  meta/classes/image_types.bbclass                         | 13 ++++++++++++-
>  meta/recipes-devtools/mtd/mtd-utils_git.bb               |  2 +-
>  meta/recipes-extended/tar/tar-replacement-native_1.28.bb | 14 +++++++++++++-
>  3 files changed, 26 insertions(+), 3 deletions(-)
> 

Re: [PATCH 0/3] preserve xattrs in images

[Patrick Ohly]

On Wed, 2015-08-12 at 09:34 -0500, Mark Hatle wrote:
> Is there something here that enables the tar-replacement-native?  Or is the user
> just expected to know they need it?

The user is supposed to ensure that it gets depended on if (and only if)
needed.

I gave the following usage instructions in the comments of the
image_types.bbclass patch:

+# By default, tar from the host is used, which can be quite old. If
+# you need special parameters (like --xattrs) which are only supported
+# by GNU tar upstream >= 1.27, then override that default:
+# IMAGE_CMD_TAR = "tar-native --xattrs --xattrs-include=*"
+# IMAGE_DEPENDS_tar_append = " tar-replacement-native"

> If you look at meta/classes/sanity.bbclass, there is already a tar check for a
> specific old version that can't handle overwriting symlinks properly.  It should
> be possible to check for the distro feature of xattr and that the host tar has
> this.  It would be good to prevent the user from having a problem early (sanity)
> rather then at filesystem generation time.

The way the patch series is intended to be used at the moment, the user
will not have a problem regardless of the host tar, because image
creation will always use a recent enough tar. It might build
tar-replacement-native unnecessarily, though.

A distro might also impose additional restrictions on the host's tar and
then will not need to be build tar-replacement-native; I haven't tried
that, because I'd still like to be compatible with Debian 7 which would
fail the requirement.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 1/3] tar-replacement-native: avoid race condition with host tar

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On 11 August 2015 at 09:44, Patrick Ohly <patrick.ohly@intel.com> wrote:

> Installing "tar" into the sysroot leads to race conditions
> (tasks which do not depend on tar-replacement-native may already
> call tar while it's installation is incomplete). Avoid those
> by installing only the tar binary under the name tar-native.
>

So the point of tar-replacement-native was that it's a *replacement* for
the host tar, so it needs the same name.  If it was to be installed as an
opt-in alternative there's a mechanism for doing that already (see
EXTRANATIVEPATH) but that would break the purpose of tar-replacement-native.

I think for this situation you should show how to enable xattrs for images,
with the caveat that if your host tar isn't suitable the user should
bitbake just tar-replacement-native *first*.

Ross

[-- Attachment #2: Type: text/html, Size: 1480 bytes --]

Re: [PATCH 3/3] mtd-utils: keep xattr support enabled

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

On 12 August 2015 at 10:33, Patrick Ohly <patrick.ohly@intel.com> wrote:

> > There's a "xattr" DISTRO_FEATURE that should be respected here for
> > target builds (and explicitly enabled for native I guess).
>
> I can add that check, if you want.
>
> > Does enabling xattr mean adding build dependencies?
>
> No, the conditional code uses lgetxattr and llistxattr, which are in
> glibc.


From five seconds of google it looks like these are in musl and uclibc too,
so I've merged this  to MUT as-is.

Ross

[-- Attachment #2: Type: text/html, Size: 946 bytes --]

Re: [PATCH 1/3] tar-replacement-native: avoid race condition with host tar

[Paul Eggleton]

On Friday 14 August 2015 11:47:16 Burton, Ross wrote:
> On 11 August 2015 at 09:44, Patrick Ohly <patrick.ohly@intel.com> wrote:
> > Installing "tar" into the sysroot leads to race conditions
> > (tasks which do not depend on tar-replacement-native may already
> > call tar while it's installation is incomplete). Avoid those
> > by installing only the tar binary under the name tar-native.
> 
> So the point of tar-replacement-native was that it's a *replacement* for
> the host tar, so it needs the same name.  If it was to be installed as an
> opt-in alternative there's a mechanism for doing that already (see
> EXTRANATIVEPATH) but that would break the purpose of tar-replacement-native.
> 
> I think for this situation you should show how to enable xattrs for images,
> with the caveat that if your host tar isn't suitable the user should
> bitbake just tar-replacement-native *first*.

Or install the buildtools perhaps.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 1/3] tar-replacement-native: avoid race condition with host tar

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On 14 August 2015 at 12:01, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> Or install the buildtools perhaps.
>

Yes, I had that in my head but someone failed to write it down.  There's a
fair chance that a host with a sufficiently old tar also needs other
updates, and that's when buildtools is used (which includes tar).

Ross

[-- Attachment #2: Type: text/html, Size: 755 bytes --]

Re: [PATCH 1/3] tar-replacement-native: avoid race condition with host tar

[Patrick Ohly]

On Fri, 2015-08-14 at 12:03 +0100, Burton, Ross wrote:
> 
> On 14 August 2015 at 12:01, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
>         Or install the buildtools perhaps.
>         
> 
> 
> Yes, I had that in my head but someone failed to write it down.
> There's a fair chance that a host with a sufficiently old tar also
> needs other updates, and that's when buildtools is used (which
> includes tar).

The hosts we talk about (Debian 7, CentOS 6) work fine for normal usage
without having to install buildtools. Having to install them for a
situation that could also be dealt with by bitbake automatically seems
not very user friendly.

On IRC, Ross suggested that I should rewrite the regular tar recipe such
that it installs tar as opt-in binary and remove the (mostly obsolete)
tar-replacement-native). I'll send updated patches based on that.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 1/3] tar-replacement-native: avoid race condition with host tar

[Paul Eggleton]

On Friday 14 August 2015 16:52:08 Patrick Ohly wrote:
> On Fri, 2015-08-14 at 12:03 +0100, Burton, Ross wrote:
> > On 14 August 2015 at 12:01, Paul Eggleton
> > 
> > <paul.eggleton@linux.intel.com> wrote:
> >         Or install the buildtools perhaps.
> > 
> > Yes, I had that in my head but someone failed to write it down.
> > There's a fair chance that a host with a sufficiently old tar also
> > needs other updates, and that's when buildtools is used (which
> > includes tar).
> 
> The hosts we talk about (Debian 7, CentOS 6) work fine for normal usage
> without having to install buildtools.

Really? You must have added *something* on top of CentOS 6 because last I 
checked its git, tar and python versions supplied out of the box were way too 
old to even get the build to start.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH v2 0/2] xattr + tar

[Patrick Ohly]

This gets rid of tar-replacement-native and instead allows installing tar
as opt-in binary under a path where it is not normally picked up.

The mtd-utils patch is not included anymore because Ross already staged it.

The following changes since commit 4e0fb630d3bb0250ece6e2071ae49b2601ae3e57:

  tzdata: update to 2015f (2015-08-14 08:30:21 +0100)

are available in the git repository at:

  git://github.com/pohly/openembedded-core xattr
  https://github.com/pohly/openembedded-core/tree/xattr

Patrick Ohly (2):
  tar-replacement-native: relocate via NATIVE_PACKAGE_PATH_SUFFIX
  image_types.bbclass: allow replacing tar command

 meta/classes/image_types.bbclass                         | 14 +++++++++++++-
 meta/recipes-extended/tar/tar-replacement-native_1.28.bb |  6 ------
 meta/recipes-extended/tar/tar.inc                        |  6 +++++-
 3 files changed, 18 insertions(+), 8 deletions(-)
 delete mode 100644 meta/recipes-extended/tar/tar-replacement-native_1.28.bb

-- 
2.1.4

[PATCH v2 1/2] tar-replacement-native: relocate via NATIVE_PACKAGE_PATH_SUFFIX

[Patrick Ohly]

Building tar-replacement-native as replacement of the host's tar in
the standard path was meant to be done manually by a user in
preparation for the regular bitbake run. Such a usage has been
superseeded by installing the pre-compiled buildutils and might have
been broken on hosts which need it by the sanity check for tar >=
1.26.

Therefore tar-replacement-native_1.28.bb can be removed in favor of
adapting the normal tar recipe such that it installs an opt-in binary
under a different path.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/recipes-extended/tar/tar-replacement-native_1.28.bb | 6 ------
 meta/recipes-extended/tar/tar.inc                        | 6 +++++-
 2 files changed, 5 insertions(+), 7 deletions(-)
 delete mode 100644 meta/recipes-extended/tar/tar-replacement-native_1.28.bb

diff --git a/meta/recipes-extended/tar/tar-replacement-native_1.28.bb b/meta/recipes-extended/tar/tar-replacement-native_1.28.bb
deleted file mode 100644
index 071ede7..0000000
--- a/meta/recipes-extended/tar/tar-replacement-native_1.28.bb
+++ /dev/null
@@ -1,6 +0,0 @@
-require tar_${PV}.bb
-
-inherit native
-
-BPN = "tar"
-EXTRAINSTALL = ""
diff --git a/meta/recipes-extended/tar/tar.inc b/meta/recipes-extended/tar/tar.inc
index 7416927..752b74d 100644
--- a/meta/recipes-extended/tar/tar.inc
+++ b/meta/recipes-extended/tar/tar.inc
@@ -18,6 +18,7 @@ acpaths = "-I ./m4"
 
 EXTRAINSTALL = "do_install_extra"
 EXTRAINSTALL_class-nativesdk = ""
+EXTRAINSTALL_class-native = ""
 
 do_install () {
     autotools_do_install
@@ -48,4 +49,7 @@ ALTERNATIVE_${PN}-rmt_class-nativesdk = ""
 ALTERNATIVE_LINK_NAME[tar] = "${base_bindir}/tar"
 ALTERNATIVE_LINK_NAME[rmt] = "${base_sbindir}/rmt"
 
-BBCLASSEXTEND = "nativesdk"
+PROVIDES_append_class-native = " tar-replacement-native"
+NATIVE_PACKAGE_PATH_SUFFIX = "/${PN}"
+
+BBCLASSEXTEND = "native nativesdk"
-- 
2.1.4

[PATCH v2 2/2] image_types.bbclass: allow replacing tar command

[Patrick Ohly]

Usually, the host's tar command is sufficient. However, special cases
like archiving xattrs depend on a modern GNU tar version. The new
IMAGE_CMD_TAR makes that possible, with xattrs given as example.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/image_types.bbclass | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index cc789fc..2f782dc 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -71,7 +71,19 @@ IMAGE_CMD_btrfs () {
 IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs ${EXTRA_IMAGECMD} -noappend"
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
-IMAGE_CMD_tar = "tar -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar -C ${IMAGE_ROOTFS} ."
+
+# By default, tar from the host is used, which can be quite old. If
+# you need special parameters (like --xattrs) which are only supported
+# by GNU tar upstream >= 1.27, then override that default:
+# IMAGE_CMD_TAR = "tar --xattrs --xattrs-include=*"
+# IMAGE_DEPENDS_tar_append = " tar-replacement-native"
+# EXTRANATIVEPATH += "tar-native"
+#
+# The GNU documentation does not specify whether --xattrs-include is necessary.
+# In practice, it turned out to be not needed when creating archives and
+# required when extracting, but it seems prudent to use it in both cases.
+IMAGE_CMD_TAR ?= "tar"
+IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar -C ${IMAGE_ROOTFS} ."
 
 do_rootfs[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
-- 
2.1.4

Re: [PATCH v2 1/2] tar-replacement-native: relocate via NATIVE_PACKAGE_PATH_SUFFIX

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

On 14 August 2015 at 15:59, Patrick Ohly <patrick.ohly@intel.com> wrote:

>  EXTRAINSTALL = "do_install_extra"
>  EXTRAINSTALL_class-nativesdk = ""
> +EXTRAINSTALL_class-native = ""
>

As this is presumably meant to be target-only logic, you could refactor
this to:

do_install() {
    autotools_do_install
    ln -s tar ${D}${bindir}/gtar
}

do_install_append_class-target() {
  (contents of do_install_extra)
}

Ross

[-- Attachment #2: Type: text/html, Size: 1184 bytes --]

Re: [PATCH 1/3] tar-replacement-native: avoid race condition with host tar

[Patrick Ohly]

On Fri, 2015-08-14 at 15:56 +0100, Paul Eggleton wrote:
> On Friday 14 August 2015 16:52:08 Patrick Ohly wrote:
> > On Fri, 2015-08-14 at 12:03 +0100, Burton, Ross wrote:
> > > On 14 August 2015 at 12:01, Paul Eggleton
> > > 
> > > <paul.eggleton@linux.intel.com> wrote:
> > >         Or install the buildtools perhaps.
> > > 
> > > Yes, I had that in my head but someone failed to write it down.
> > > There's a fair chance that a host with a sufficiently old tar also
> > > needs other updates, and that's when buildtools is used (which
> > > includes tar).
> > 
> > The hosts we talk about (Debian 7, CentOS 6) work fine for normal usage
> > without having to install buildtools.
> 
> Really? You must have added *something* on top of CentOS 6 because last I 
> checked its git, tar and python versions supplied out of the box were way too 
> old to even get the build to start.

I thought someone had said that CentOS is okay out-of-the-box, but I may
have misunderstood.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

[PATCH v3 0/2] xattr + tar

[Patrick Ohly]

This time with tar's do_install logic refactored as suggested by Ross.

The following changes since commit 4e0fb630d3bb0250ece6e2071ae49b2601ae3e57:

  tzdata: update to 2015f (2015-08-14 08:30:21 +0100)

are available in the git repository at:

  git://github.com/pohly/openembedded-core xattr
  https://github.com/pohly/openembedded-core/tree/xattr

Patrick Ohly (2):
  tar-replacement-native: relocate via NATIVE_PACKAGE_PATH_SUFFIX
  image_types.bbclass: allow replacing tar command

 meta/classes/image_types.bbclass                         | 14 +++++++++++++-
 meta/recipes-extended/tar/tar-replacement-native_1.28.bb |  6 ------
 meta/recipes-extended/tar/tar.inc                        | 11 +++++------
 3 files changed, 18 insertions(+), 13 deletions(-)
 delete mode 100644 meta/recipes-extended/tar/tar-replacement-native_1.28.bb

-- 
2.1.4

[PATCH v3 1/2] tar-replacement-native: relocate via NATIVE_PACKAGE_PATH_SUFFIX

[Patrick Ohly]

Building tar-replacement-native as replacement of the host's tar in
the standard path was meant to be done manually by a user in
preparation for the regular bitbake run. Such a usage has been
superseeded by installing the pre-compiled buildutils and might have
been broken on hosts which need it by the sanity check for tar >=
1.26.

Therefore tar-replacement-native_1.28.bb can be removed in favor of
adapting the normal tar recipe such that it installs an opt-in binary
under a different path.

The special do_install logic is explicitly limited to class-target,
instead of making it the default and disabling it (which would be the
case for class-native and class-nativesdk).

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/recipes-extended/tar/tar-replacement-native_1.28.bb |  6 ------
 meta/recipes-extended/tar/tar.inc                        | 11 +++++------
 2 files changed, 5 insertions(+), 12 deletions(-)
 delete mode 100644 meta/recipes-extended/tar/tar-replacement-native_1.28.bb

diff --git a/meta/recipes-extended/tar/tar-replacement-native_1.28.bb b/meta/recipes-extended/tar/tar-replacement-native_1.28.bb
deleted file mode 100644
index 071ede7..0000000
--- a/meta/recipes-extended/tar/tar-replacement-native_1.28.bb
+++ /dev/null
@@ -1,6 +0,0 @@
-require tar_${PV}.bb
-
-inherit native
-
-BPN = "tar"
-EXTRAINSTALL = ""
diff --git a/meta/recipes-extended/tar/tar.inc b/meta/recipes-extended/tar/tar.inc
index 7416927..b339c43 100644
--- a/meta/recipes-extended/tar/tar.inc
+++ b/meta/recipes-extended/tar/tar.inc
@@ -16,16 +16,12 @@ EXTRA_OECONF += "DEFAULT_RMT_DIR=${base_sbindir}"
 # directory.
 acpaths = "-I ./m4"
 
-EXTRAINSTALL = "do_install_extra"
-EXTRAINSTALL_class-nativesdk = ""
-
 do_install () {
     autotools_do_install
     ln -s tar ${D}${bindir}/gtar
-    ${EXTRAINSTALL}
 }
 
-do_install_extra () {
+do_install_append_class-target() {
     install -d ${D}${base_bindir}
     mv ${D}${bindir}/tar ${D}${base_bindir}/tar
     mv ${D}${bindir}/gtar ${D}${base_bindir}/gtar
@@ -48,4 +44,7 @@ ALTERNATIVE_${PN}-rmt_class-nativesdk = ""
 ALTERNATIVE_LINK_NAME[tar] = "${base_bindir}/tar"
 ALTERNATIVE_LINK_NAME[rmt] = "${base_sbindir}/rmt"
 
-BBCLASSEXTEND = "nativesdk"
+PROVIDES_append_class-native = " tar-replacement-native"
+NATIVE_PACKAGE_PATH_SUFFIX = "/${PN}"
+
+BBCLASSEXTEND = "native nativesdk"
-- 
2.1.4

[PATCH v3 2/2] image_types.bbclass: allow replacing tar command

[Patrick Ohly]

Usually, the host's tar command is sufficient. However, special cases
like archiving xattrs depend on a modern GNU tar version. The new
IMAGE_CMD_TAR makes that possible, with xattrs given as example.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/image_types.bbclass | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index cc789fc..2f782dc 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -71,7 +71,19 @@ IMAGE_CMD_btrfs () {
 IMAGE_CMD_squashfs = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs ${EXTRA_IMAGECMD} -noappend"
 IMAGE_CMD_squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD_squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
-IMAGE_CMD_tar = "tar -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar -C ${IMAGE_ROOTFS} ."
+
+# By default, tar from the host is used, which can be quite old. If
+# you need special parameters (like --xattrs) which are only supported
+# by GNU tar upstream >= 1.27, then override that default:
+# IMAGE_CMD_TAR = "tar --xattrs --xattrs-include=*"
+# IMAGE_DEPENDS_tar_append = " tar-replacement-native"
+# EXTRANATIVEPATH += "tar-native"
+#
+# The GNU documentation does not specify whether --xattrs-include is necessary.
+# In practice, it turned out to be not needed when creating archives and
+# required when extracting, but it seems prudent to use it in both cases.
+IMAGE_CMD_TAR ?= "tar"
+IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cvf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.tar -C ${IMAGE_ROOTFS} ."
 
 do_rootfs[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD_cpio () {
-- 
2.1.4

Re: [PATCH 1/3] tar-replacement-native: avoid race condition with host tar

[Paul Eggleton]

On Friday 14 August 2015 17:38:38 Patrick Ohly wrote:
> On Fri, 2015-08-14 at 15:56 +0100, Paul Eggleton wrote:
> > On Friday 14 August 2015 16:52:08 Patrick Ohly wrote:
> > > On Fri, 2015-08-14 at 12:03 +0100, Burton, Ross wrote:
> > > > On 14 August 2015 at 12:01, Paul Eggleton
> > > > 
> > > > <paul.eggleton@linux.intel.com> wrote:
> > > >         Or install the buildtools perhaps.
> > > > 
> > > > Yes, I had that in my head but someone failed to write it down.
> > > > There's a fair chance that a host with a sufficiently old tar also
> > > > needs other updates, and that's when buildtools is used (which
> > > > includes tar).
> > > 
> > > The hosts we talk about (Debian 7, CentOS 6) work fine for normal usage
> > > without having to install buildtools.
> > 
> > Really? You must have added *something* on top of CentOS 6 because last I
> > checked its git, tar and python versions supplied out of the box were way
> > too old to even get the build to start.
> 
> I thought someone had said that CentOS is okay out-of-the-box, but I may
> have misunderstood.

CentOS 7 might be though since it's quite a bit newer.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 3/3] mtd-utils: keep xattr support enabled

[Patrick Ohly]

On Fri, 2015-08-14 at 11:51 +0100, Burton, Ross wrote:
> 
> On 12 August 2015 at 10:33, Patrick Ohly <patrick.ohly@intel.com>
> wrote:
>         > There's a "xattr" DISTRO_FEATURE that should be respected
>         here for
>         > target builds (and explicitly enabled for native I guess).
>         
>         I can add that check, if you want.
>         
>         > Does enabling xattr mean adding build dependencies?
>         
>         No, the conditional code uses lgetxattr and llistxattr, which
>         are in
>         glibc.
> 
> 
> From five seconds of google it looks like these are in musl and uclibc
> too, so I've merged this  to MUT as-is.

However, it turned out that the XATTR code also introduces a dependency
on sys/acl.h:

mkfs.jffs2.c:

#ifndef WITHOUT_XATTR
#include <sys/xattr.h>
#include <sys/acl.h>
#endif

sys/acl.h gets provided by acl, so it may or may not work depending on
build order and files on the host.

I hadn't caught that because I had only looked at library dependencies,
sorry for that.

What's the right fix? Add the dependency on acl-native in the native
case, disable XATTR support for target builds (same state as before) or
make it somehow depend on both "xattr" and "acl" distro features?

Or only check for "xattr"? I'm uncertain how the two are related.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 3/3] mtd-utils: keep xattr support enabled

[Mark Hatle]

On 8/25/15 6:46 AM, Patrick Ohly wrote:
> On Fri, 2015-08-14 at 11:51 +0100, Burton, Ross wrote:
>>
>> On 12 August 2015 at 10:33, Patrick Ohly <patrick.ohly@intel.com>
>> wrote:
>>         > There's a "xattr" DISTRO_FEATURE that should be respected
>>         here for
>>         > target builds (and explicitly enabled for native I guess).
>>         
>>         I can add that check, if you want.
>>         
>>         > Does enabling xattr mean adding build dependencies?
>>         
>>         No, the conditional code uses lgetxattr and llistxattr, which
>>         are in
>>         glibc.
>>
>>
>> From five seconds of google it looks like these are in musl and uclibc
>> too, so I've merged this  to MUT as-is.
> 
> However, it turned out that the XATTR code also introduces a dependency
> on sys/acl.h:
> 
> mkfs.jffs2.c:
> 
> #ifndef WITHOUT_XATTR
> #include <sys/xattr.h>
> #include <sys/acl.h>
> #endif
> 
> sys/acl.h gets provided by acl, so it may or may not work depending on
> build order and files on the host.
> 
> I hadn't caught that because I had only looked at library dependencies,
> sorry for that.
> 
> What's the right fix? Add the dependency on acl-native in the native
> case, disable XATTR support for target builds (same state as before) or
> make it somehow depend on both "xattr" and "acl" distro features?
> 
> Or only check for "xattr"? I'm uncertain how the two are related.
> 

Use PACKAGECONFIG, add xattr as a configuration that in turn adds the 'acl' as a
dependency.  Then acl-native will automatically be converted from 'acl' if the
'-native' version is built.

This is typically how I've seen it implemented in other places.  (Not necessary
xattr.. but with the push for more Linux Security Models coming from users and
other layers, I think xattr is something we need to make sure really works
properly.)

--Mark

Re: [PATCH 3/3] mtd-utils: keep xattr support enabled

[Andrea Adami]

[-- Attachment #1: Type: text/plain, Size: 2423 bytes --]

Patrick,

sorry I didn't put you in cc.
My quick&dirty workaround: http://patchwork.openembedded.org/patch/101169/

Cheers

Andrea

On Tue, Aug 25, 2015 at 3:26 PM, Mark Hatle <mark.hatle@windriver.com>
wrote:

> On 8/25/15 6:46 AM, Patrick Ohly wrote:
> > On Fri, 2015-08-14 at 11:51 +0100, Burton, Ross wrote:
> >>
> >> On 12 August 2015 at 10:33, Patrick Ohly <patrick.ohly@intel.com>
> >> wrote:
> >>         > There's a "xattr" DISTRO_FEATURE that should be respected
> >>         here for
> >>         > target builds (and explicitly enabled for native I guess).
> >>
> >>         I can add that check, if you want.
> >>
> >>         > Does enabling xattr mean adding build dependencies?
> >>
> >>         No, the conditional code uses lgetxattr and llistxattr, which
> >>         are in
> >>         glibc.
> >>
> >>
> >> From five seconds of google it looks like these are in musl and uclibc
> >> too, so I've merged this  to MUT as-is.
> >
> > However, it turned out that the XATTR code also introduces a dependency
> > on sys/acl.h:
> >
> > mkfs.jffs2.c:
> >
> > #ifndef WITHOUT_XATTR
> > #include <sys/xattr.h>
> > #include <sys/acl.h>
> > #endif
> >
> > sys/acl.h gets provided by acl, so it may or may not work depending on
> > build order and files on the host.
> >
> > I hadn't caught that because I had only looked at library dependencies,
> > sorry for that.
> >
> > What's the right fix? Add the dependency on acl-native in the native
> > case, disable XATTR support for target builds (same state as before) or
> > make it somehow depend on both "xattr" and "acl" distro features?
> >
> > Or only check for "xattr"? I'm uncertain how the two are related.
> >
>
> Use PACKAGECONFIG, add xattr as a configuration that in turn adds the
> 'acl' as a
> dependency.  Then acl-native will automatically be converted from 'acl' if
> the
> '-native' version is built.
>
> This is typically how I've seen it implemented in other places.  (Not
> necessary
> xattr.. but with the push for more Linux Security Models coming from users
> and
> other layers, I think xattr is something we need to make sure really works
> properly.)
>
> --Mark
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3664 bytes --]

Re: [PATCH 3/3] mtd-utils: keep xattr support enabled

[Patrick Ohly]

On Tue, 2015-08-25 at 08:26 -0500, Mark Hatle wrote:
> On 8/25/15 6:46 AM, Patrick Ohly wrote:
> > On Fri, 2015-08-14 at 11:51 +0100, Burton, Ross wrote:
> >>
> >> On 12 August 2015 at 10:33, Patrick Ohly <patrick.ohly@intel.com>
> >> wrote:
> >>         > There's a "xattr" DISTRO_FEATURE that should be respected
> >>         here for
> >>         > target builds (and explicitly enabled for native I guess).
> >>         
> >>         I can add that check, if you want.
> >>         
> >>         > Does enabling xattr mean adding build dependencies?
> >>         
> >>         No, the conditional code uses lgetxattr and llistxattr, which
> >>         are in
> >>         glibc.
> >>
> >>
> >> From five seconds of google it looks like these are in musl and uclibc
> >> too, so I've merged this  to MUT as-is.
> > 
> > However, it turned out that the XATTR code also introduces a dependency
> > on sys/acl.h:
> > 
> > mkfs.jffs2.c:
> > 
> > #ifndef WITHOUT_XATTR
> > #include <sys/xattr.h>
> > #include <sys/acl.h>
> > #endif
> > 
> > sys/acl.h gets provided by acl, so it may or may not work depending on
> > build order and files on the host.
> > 
> > I hadn't caught that because I had only looked at library dependencies,
> > sorry for that.
> > 
> > What's the right fix? Add the dependency on acl-native in the native
> > case, disable XATTR support for target builds (same state as before) or
> > make it somehow depend on both "xattr" and "acl" distro features?
> > 
> > Or only check for "xattr"? I'm uncertain how the two are related.
> > 
> 
> Use PACKAGECONFIG, add xattr as a configuration that in turn adds the 'acl' as a
> dependency.  Then acl-native will automatically be converted from 'acl' if the
> '-native' version is built.

I agree, a PACKAGECONFIG sounds like the right approach. However, what
should the default depend on? The "xattr" or "acl" distro feature? What
if someone wants xattr, but not acl? Should xattr support in mtd-utils
be enabled in that case even though it pulls in the acl recipe as
dependency?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 3/3] mtd-utils: keep xattr support enabled

[Mark Hatle]

On 8/25/15 10:27 AM, Patrick Ohly wrote:
> On Tue, 2015-08-25 at 08:26 -0500, Mark Hatle wrote:
>> On 8/25/15 6:46 AM, Patrick Ohly wrote:
>>> On Fri, 2015-08-14 at 11:51 +0100, Burton, Ross wrote:
>>>>
>>>> On 12 August 2015 at 10:33, Patrick Ohly <patrick.ohly@intel.com>
>>>> wrote:
>>>>         > There's a "xattr" DISTRO_FEATURE that should be respected
>>>>         here for
>>>>         > target builds (and explicitly enabled for native I guess).
>>>>         
>>>>         I can add that check, if you want.
>>>>         
>>>>         > Does enabling xattr mean adding build dependencies?
>>>>         
>>>>         No, the conditional code uses lgetxattr and llistxattr, which
>>>>         are in
>>>>         glibc.
>>>>
>>>>
>>>> From five seconds of google it looks like these are in musl and uclibc
>>>> too, so I've merged this  to MUT as-is.
>>>
>>> However, it turned out that the XATTR code also introduces a dependency
>>> on sys/acl.h:
>>>
>>> mkfs.jffs2.c:
>>>
>>> #ifndef WITHOUT_XATTR
>>> #include <sys/xattr.h>
>>> #include <sys/acl.h>
>>> #endif
>>>
>>> sys/acl.h gets provided by acl, so it may or may not work depending on
>>> build order and files on the host.
>>>
>>> I hadn't caught that because I had only looked at library dependencies,
>>> sorry for that.
>>>
>>> What's the right fix? Add the dependency on acl-native in the native
>>> case, disable XATTR support for target builds (same state as before) or
>>> make it somehow depend on both "xattr" and "acl" distro features?
>>>
>>> Or only check for "xattr"? I'm uncertain how the two are related.
>>>
>>
>> Use PACKAGECONFIG, add xattr as a configuration that in turn adds the 'acl' as a
>> dependency.  Then acl-native will automatically be converted from 'acl' if the
>> '-native' version is built.
> 
> I agree, a PACKAGECONFIG sounds like the right approach. However, what
> should the default depend on? The "xattr" or "acl" distro feature? What
> if someone wants xattr, but not acl? Should xattr support in mtd-utils
> be enabled in that case even though it pulls in the acl recipe as
> dependency?
> 

I think that depends entirely on the dependencies in the package.  I've seen a
few things that do acl w/o requiring xattr.. but they're usually linked together.

It sounds like at a minimum xattr should be the PACKAGECONFIG based on the xattr
distro feature.

--Mark