[PATCH v3 0/8] Fix for #7040 - Support for /usr merge

[PATCH v3 0/8] Fix for #7040 - Support for /usr merge

[Amarnath Valluri]

This set of patches implements the long-standing feature - merging of /bin,
/sbin, /libs to their /usr counterparts.

It introduces a 'usrmerge' DISTRO_FEATURE, when it's enabled:
    - other than native all base paths(base_{bin,sbin,lib}dir) points to
      ${exec_prefix}/{bin,sbin,lib}
    - base-files package adds the needed root links(/bin, /sbin, /lib{32,64})
      points to /usr counterparts
    - As most of shell scripts refer "#!/bin/{sh,bash}" inside script, so makes
      sure that get added to RPROVIDES_ list by those packages to satisfy the
      build dependencies.

The assumption is that no recipe uses hard coded paths in recipes where
pre-defined bitbake configuration variables are available. I have fixed quite
many recipes that i found which violates this assumption. A sanity check is
added for reporting these broken recipies.

I have tested this change on core-minimal, sato, windriver cube-desktop and
intel-iot-refkit image builds.  And i would prefer to run autobuilder tests
prior to merging.

Changes in v2:
    - Moved root links creation out from 'base-files' to pre-rootfs.
    - Fixes to lsb, lttng-modules recipes., that are failed in autobuilder
      tests.
    - Added insane check to usrmerge feature.
    - Fixed typos in commit messages.

Changes in v3:
    - Rebased against master and dropped patches those got already merged.
    - As per review comments added new configuration variable ${root_prefix}
      that will be altered based on 'usrmerge' distro feature.
    - Made use of ${root_prefix} in place of ${base_preix} both in systemd
      recipe and cross.bbclass
    - Dropped patch that is touching nativesdk configuration.


Amarnath Valluri (8):
  bitbake.conf: support for merged usr with DISTRO_FEATURE usrmerge
  cross.bbclass: merged /usr support
  systemd: chagnes to support merged /usr
  bash: changes to support merged /usr
  busybox: changes to support merged /usr
  sed: changes to support merged /usr
  image: create symlinks needed for merged /usr
  insane.bbclass: Add package QA check for merged /usr.

 meta/classes/cross.bbclass               |  4 ++--
 meta/classes/image.bbclass               | 23 +++++++++++++++++++++++
 meta/classes/insane.bbclass              | 16 ++++++++++++++++
 meta/conf/bitbake.conf                   | 10 ++++++----
 meta/recipes-core/busybox/busybox.inc    |  2 ++
 meta/recipes-core/systemd/systemd_232.bb |  6 +++---
 meta/recipes-extended/bash/bash.inc      |  2 ++
 meta/recipes-extended/sed/sed_4.2.2.bb   |  1 +
 8 files changed, 55 insertions(+), 9 deletions(-)

-- 
2.7.4

[PATCH v3 1/8] bitbake.conf: support for merged usr with DISTRO_FEATURE usrmerge

[Amarnath Valluri]

A new configuration variable ${root_prefix} added, which shall be used by all
base_{lib,bin,sbin}dir variables. When usrmerge DISTRO_FEATURE is enabled
${root_prefix} points to ${exec_prefix} otherwise to ${base_prefix}

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/conf/bitbake.conf | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 3ad905c..1d3c79f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -17,11 +17,13 @@ export base_prefix = ""
 export prefix = "/usr"
 export exec_prefix = "${prefix}"
 
+export root_prefix = "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', '${exec_prefix}', '${base_prefix}', d)}"
+
 # Base paths
-export base_bindir = "${base_prefix}/bin"
-export base_sbindir = "${base_prefix}/sbin"
-export base_libdir = "${base_prefix}/${baselib}"
-export nonarch_base_libdir = "${base_prefix}/lib"
+export base_bindir = "${root_prefix}/bin"
+export base_sbindir = "${root_prefix}/sbin"
+export base_libdir = "${root_prefix}/${baselib}"
+export nonarch_base_libdir = "${root_prefix}/lib"
 
 # Architecture independent paths
 export sysconfdir = "${base_prefix}/etc"
-- 
2.7.4

[PATCH v3 2/8] cross.bbclass: merged /usr support

[Amarnath Valluri]

To be align with base configuration change, use ${root_prefix} while preparing
${target_base_libdir}.

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/classes/cross.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/cross.bbclass b/meta/classes/cross.bbclass
index 8757303..7720185 100644
--- a/meta/classes/cross.bbclass
+++ b/meta/classes/cross.bbclass
@@ -50,10 +50,10 @@ SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"
 # Path mangling needed by the cross packaging
 # Note that we use := here to ensure that libdir and includedir are
 # target paths.
-target_base_prefix := "${base_prefix}"
+target_root_prefix := "${root_prefix}"
 target_prefix := "${prefix}"
 target_exec_prefix := "${exec_prefix}"
-target_base_libdir = "${target_base_prefix}/${baselib}"
+target_base_libdir = "${target_root_prefix}/${baselib}"
 target_libdir = "${target_exec_prefix}/${baselib}"
 target_includedir := "${includedir}"
 
-- 
2.7.4

Re: [PATCH v3 2/8] cross.bbclass: merged /usr support

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Amarnath Valluri
> Sent: den 13 juni 2017 08:53
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH v3 2/8] cross.bbclass: merged /usr support
> 
> To be align with base configuration change, use ${root_prefix} while
> preparing
> ${target_base_libdir}.
> 
> Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
> ---
>  meta/classes/cross.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/cross.bbclass b/meta/classes/cross.bbclass
> index 8757303..7720185 100644
> --- a/meta/classes/cross.bbclass
> +++ b/meta/classes/cross.bbclass
> @@ -50,10 +50,10 @@ SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"
>  # Path mangling needed by the cross packaging
>  # Note that we use := here to ensure that libdir and includedir are
>  # target paths.
> -target_base_prefix := "${base_prefix}"
> +target_root_prefix := "${root_prefix}"

It is probably better to do:

target_base_prefix := "${root_prefix}"

I.e., maintain the target_base_prefix variable, in case it is used 
in some external layer.

>  target_prefix := "${prefix}"
>  target_exec_prefix := "${exec_prefix}"
> -target_base_libdir = "${target_base_prefix}/${baselib}"
> +target_base_libdir = "${target_root_prefix}/${baselib}"
>  target_libdir = "${target_exec_prefix}/${baselib}"
>  target_includedir := "${includedir}"
> 
> --
> 2.7.4

//Peter

[PATCH v4 2/8] cross.bbclass: merged /usr support

[Amarnath Valluri]

Use ${root_prefix} instead of ${base_prefix} while setting
${target_base_prefix}, otherwise we might loose the root prefix configuration
change in case of 'usrmerge' distro feature is enabled.

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/classes/cross.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/cross.bbclass b/meta/classes/cross.bbclass
index 8757303..4887317 100644
--- a/meta/classes/cross.bbclass
+++ b/meta/classes/cross.bbclass
@@ -50,7 +50,7 @@ SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"
 # Path mangling needed by the cross packaging
 # Note that we use := here to ensure that libdir and includedir are
 # target paths.
-target_base_prefix := "${base_prefix}"
+target_base_prefix := "${root_prefix}"
 target_prefix := "${prefix}"
 target_exec_prefix := "${exec_prefix}"
 target_base_libdir = "${target_base_prefix}/${baselib}"
-- 
2.7.4

[PATCH v3 3/8] systemd: chagnes to support merged /usr

[Amarnath Valluri]

- Disable the split-usr support in systemd when 'usrmerge' DISTRO_FEATURE is
  enabled.
- Modify rootprefix to point to ${root_prefix}, rather than ${base_prefix}.
- And fixed firmware path to use ${nonarch_base_libdir} instead of hard-coded
  '/lib', because when 'usrmege' distro feature enabled this path would be
  '/usr/lib'.

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/recipes-core/systemd/systemd_232.bb | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_232.bb b/meta/recipes-core/systemd/systemd_232.bb
index f843c58..22c7254 100644
--- a/meta/recipes-core/systemd/systemd_232.bb
+++ b/meta/recipes-core/systemd/systemd_232.bb
@@ -131,7 +131,7 @@ CACHED_CONFIGUREVARS += "ac_cv_path_SULOGIN=${base_sbindir}/sulogin"
 
 # Helper variables to clarify locations.  This mirrors the logic in systemd's
 # build system.
-rootprefix ?= "${base_prefix}"
+rootprefix ?= "${root_prefix}"
 rootlibdir ?= "${base_libdir}"
 rootlibexecdir = "${rootprefix}/lib"
 
@@ -149,10 +149,10 @@ CACHED_CONFIGUREVARS_class-target = "\
 EXTRA_OECONF = " --with-rootprefix=${rootprefix} \
                  --with-rootlibdir=${rootlibdir} \
                  --with-roothomedir=${ROOT_HOME} \
-                 --enable-split-usr \
+                 ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', '--disable-split-usr', '--enable-split-usr', d)} \
                  --without-python \
                  --with-sysvrcnd-path=${sysconfdir} \
-                 --with-firmware-path=/lib/firmware \
+                 --with-firmware-path=${nonarch_base_libdir}/firmware \
                  --with-testdir=${PTEST_PATH} \
                "
 # per the systemd README, define VALGRIND=1 to run under valgrind
-- 
2.7.4

Re: [PATCH v3 3/8] systemd: chagnes to support merged /usr

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Amarnath Valluri
> Sent: den 13 juni 2017 08:53
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH v3 3/8] systemd: chagnes to support merged

Change "chagnes" to "changes".

> /usr
> 
> - Disable the split-usr support in systemd when 'usrmerge' DISTRO_FEATURE is
>   enabled.
> - Modify rootprefix to point to ${root_prefix}, rather than ${base_prefix}.
> - And fixed firmware path to use ${nonarch_base_libdir} instead of hard-coded
>   '/lib', because when 'usrmege' distro feature enabled this path would be
>   '/usr/lib'.
> 
> Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
> ---
>  meta/recipes-core/systemd/systemd_232.bb | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/recipes-core/systemd/systemd_232.bb b/meta/recipes-
> core/systemd/systemd_232.bb
> index f843c58..22c7254 100644
> --- a/meta/recipes-core/systemd/systemd_232.bb
> +++ b/meta/recipes-core/systemd/systemd_232.bb
> @@ -131,7 +131,7 @@ CACHED_CONFIGUREVARS +=
> "ac_cv_path_SULOGIN=${base_sbindir}/sulogin"
> 
>  # Helper variables to clarify locations.  This mirrors the logic in systemd's
>  # build system.
> -rootprefix ?= "${base_prefix}"
> +rootprefix ?= "${root_prefix}"
>  rootlibdir ?= "${base_libdir}"
>  rootlibexecdir = "${rootprefix}/lib"
> 
> @@ -149,10 +149,10 @@ CACHED_CONFIGUREVARS_class-target = "\
>  EXTRA_OECONF = " --with-rootprefix=${rootprefix} \
>                   --with-rootlibdir=${rootlibdir} \
>                   --with-roothomedir=${ROOT_HOME} \
> -                 --enable-split-usr \
> +                 ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', '--disable-split-usr', '--enable-split-usr', d)} \

I recommend that you add a PACKAGECONFIG for this instead:

PACKAGECONFIG[usrmerge] = "--disable-split-usr,--enable-split-usr"

And add 'usrmerge' to the bb.utils.filter() call in the definition of 
PACKAGECONFIG.

>                   --without-python \
>                   --with-sysvrcnd-path=${sysconfdir} \
> -                 --with-firmware-path=/lib/firmware \
> +                 --with-firmware-path=${nonarch_base_libdir}/firmware \
>                   --with-testdir=${PTEST_PATH} \
>                 "
>  # per the systemd README, define VALGRIND=1 to run under valgrind
> --
> 2.7.4

//Peter

[PATCH v4 3/8] systemd: changes to support merged /usr

[Amarnath Valluri]

- Enable/disable the split-usr support in systemd based on 'usrmerge'
  DISTRO_FEATURE.
- Modify rootprefix to point to ${root_prefix}, rather than ${base_prefix}.
- And fixed firmware path to use ${nonarch_base_libdir} instead of hard-coded
  '/lib', because when 'usrmege' distro feature enabled this path would be
  '/usr/lib'.

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/recipes-core/systemd/systemd_232.bb | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_232.bb b/meta/recipes-core/systemd/systemd_232.bb
index f843c58..d3903a6 100644
--- a/meta/recipes-core/systemd/systemd_232.bb
+++ b/meta/recipes-core/systemd/systemd_232.bb
@@ -41,7 +41,7 @@ SRC_URI_append_libc-uclibc = "\
 SRC_URI_append_qemuall = " file://0001-core-device.c-Change-the-default-device-timeout-to-2.patch"
 
 PACKAGECONFIG ??= "xz \
-                   ${@bb.utils.filter('DISTRO_FEATURES', 'efi pam selinux ldconfig', d)} \
+                   ${@bb.utils.filter('DISTRO_FEATURES', 'efi pam selinux ldconfig usrmerge', d)} \
                    ${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'xkbcommon', '', d)} \
                    ${@bb.utils.contains('DISTRO_FEATURES', 'wifi', 'rfkill', '', d)} \
                    binfmt \
@@ -122,6 +122,7 @@ PACKAGECONFIG[bzip2] = "--enable-bzip2,--disable-bzip2,bzip2"
 PACKAGECONFIG[lz4] = "--enable-lz4,--disable-lz4,lz4"
 PACKAGECONFIG[xz] = "--enable-xz,--disable-xz,xz"
 PACKAGECONFIG[zlib] = "--enable-zlib,--disable-zlib,zlib"
+PACKAGECONFIG[usrmerge] = "--disable-split-usr, --enable-split-usr"
 
 CACHED_CONFIGUREVARS += "ac_cv_path_KILL=${base_bindir}/kill"
 CACHED_CONFIGUREVARS += "ac_cv_path_KMOD=${base_bindir}/kmod"
@@ -131,7 +132,7 @@ CACHED_CONFIGUREVARS += "ac_cv_path_SULOGIN=${base_sbindir}/sulogin"
 
 # Helper variables to clarify locations.  This mirrors the logic in systemd's
 # build system.
-rootprefix ?= "${base_prefix}"
+rootprefix ?= "${root_prefix}"
 rootlibdir ?= "${base_libdir}"
 rootlibexecdir = "${rootprefix}/lib"
 
@@ -149,10 +150,9 @@ CACHED_CONFIGUREVARS_class-target = "\
 EXTRA_OECONF = " --with-rootprefix=${rootprefix} \
                  --with-rootlibdir=${rootlibdir} \
                  --with-roothomedir=${ROOT_HOME} \
-                 --enable-split-usr \
                  --without-python \
                  --with-sysvrcnd-path=${sysconfdir} \
-                 --with-firmware-path=/lib/firmware \
+                 --with-firmware-path=${nonarch_base_libdir}/firmware \
                  --with-testdir=${PTEST_PATH} \
                "
 # per the systemd README, define VALGRIND=1 to run under valgrind
-- 
2.7.4

[PATCH v3 4/8] bash: changes to support merged /usr

[Amarnath Valluri]

Most of shell scripts refer to '#!/bin/{sh,bash}' inside the script. But when
'usrmege' feature is enabled this path will be /usr/bin/{sh, bash}.

so to satisify build dependency add '/bin/{sh,bash}' to its providers list.

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/recipes-extended/bash/bash.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-extended/bash/bash.inc b/meta/recipes-extended/bash/bash.inc
index 3e9c662..6c94a24 100644
--- a/meta/recipes-extended/bash/bash.inc
+++ b/meta/recipes-extended/bash/bash.inc
@@ -69,3 +69,5 @@ pkg_postrm_${PN} () {
 PACKAGES += "${PN}-bashbug"
 FILES_${PN} = "${bindir}/bash ${base_bindir}/bash.bash"
 FILES_${PN}-bashbug = "${bindir}/bashbug"
+
+RPROVIDES_${PN} += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', '/bin/sh /bin/bash', '', d)}"
-- 
2.7.4

Re: [PATCH v3 4/8] bash: changes to support merged /usr

[Anton Eliasson]

On Tue, 13 Jun 2017 09:52:36 +0300, Amarnath Valluri wrote:

> Most of shell scripts refer to '#!/bin/{sh,bash}' inside the script. But
> when 'usrmege' feature is enabled this path will be /usr/bin/{sh, bash}.
usrmege -> usrmerge

Also in next patch.
> 
> so to satisify build dependency add '/bin/{sh,bash}' to its providers
> list.
> 
> Signed-off-by: Amarnath Valluri
> <amarnath.valluri@intel.com>
> ---

Best Regards, Anton Eliasson

[PATCH v3 5/8] busybox: changes to support merged /usr

[Amarnath Valluri]

Most of the shell scripts refer to /bin/sh inside the script. When 'usrmege'
feature is enabled, this path would be /usr/bin/sh. Hence, to satisfy build
dependency add '/bin/sh' to it's providers list.

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/recipes-core/busybox/busybox.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
index b23eaa1..0816df6 100644
--- a/meta/recipes-core/busybox/busybox.inc
+++ b/meta/recipes-core/busybox/busybox.inc
@@ -439,3 +439,5 @@ pkg_prerm_${PN}-syslog () {
 		fi
 	fi
 }
+
+RPROVIDES_${PN} += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', '/bin/sh', '', d)}"
-- 
2.7.4

[PATCH v3 6/8] sed: changes to support merged /usr

[Amarnath Valluri]

Few of the perl scripts referring '#!/bin/sed' inside the script. But when
'usrmerge' feature is enabled this path would be /usr/bin/sed. So to satisfy
build dependency add '/bin/sed' to it's providers list.

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/recipes-extended/sed/sed_4.2.2.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-extended/sed/sed_4.2.2.bb b/meta/recipes-extended/sed/sed_4.2.2.bb
index 5aa7d8a..e31bec2 100644
--- a/meta/recipes-extended/sed/sed_4.2.2.bb
+++ b/meta/recipes-extended/sed/sed_4.2.2.bb
@@ -44,3 +44,4 @@ do_install_ptest() {
 	oe_runmake -C ${TESTDIR} install-ptest BUILDDIR=${B} DESTDIR=${D}${PTEST_PATH} TESTDIR=${TESTDIR}
 }
 
+RPROVIDES_${PN} += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', '/bin/sed', '', d)}"
-- 
2.7.4

[PATCH v3 7/8] image: create symlinks needed for merged /usr

[Amarnath Valluri]

Prepare the symlinks required for merged /usr at the time of rootfs creation.

The links created in rootfs are:
/bin --> /usr/sbin
/sbin --> /usr/sbin
/lib --> /usr/lib
/lib64 --> /usr/lib64

We cannot make these symlinks as part of 'base-files' or some other package.
Because at rootfs creation, installation of the package(say kernel) that depends
on these root folders/links fails, if package manager installs this package
prior to base-files.

These symbolic links in top level folder should present as long as
 - kerenl tools use /lib/{module,firmware}
 - shell scripts uses "#!/bin/sh"

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/classes/image.bbclass | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 85f6b9a..f08e3e1 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -617,3 +617,26 @@ do_bundle_initramfs () {
 	:
 }
 addtask bundle_initramfs after do_image_complete
+
+# Prepare the root links to point to the /usr counterparts.
+create_merged_usr_symlinks() {
+    install -m 0755 -d ${IMAGE_ROOTFS}/${base_bindir}
+    install -m 0755 -d ${IMAGE_ROOTFS}/${base_sbindir}
+    install -m 0755 -d ${IMAGE_ROOTFS}/${base_libdir}
+    lnr ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}/bin
+    lnr ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}/sbin
+    lnr ${IMAGE_ROOTFS}${base_libdir} ${IMAGE_ROOTFS}/${baselib}
+
+    if [ "${nonarch_base_libdir}" != "${base_libdir}" ]; then
+       install -m 0755 -d ${IMAGE_ROOTFS}/${nonarch_base_libdir}
+       lnr ${IMAGE_ROOTFS}${nonarch_base_libdir} ${IMAGE_ROOTFS}/lib
+    fi
+
+    # create base links for multilibs
+    multi_libdirs="${@d.getVar('MULTILIB_VARIANTS')}"
+    for d in $multi_libdirs; do
+        install -m 0755 -d ${IMAGE_ROOTFS}/${exec_prefix}/$d
+        lnr ${IMAGE_ROOTFS}/${exec_prefix}/$d ${IMAGE_ROOTFS}/$d
+    done
+}
+ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks; ', '',d)}"
-- 
2.7.4

Re: [PATCH v3 7/8] image: create symlinks needed for merged /usr

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Amarnath Valluri
> Sent: den 13 juni 2017 08:53
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH v3 7/8] image: create symlinks needed for
> merged /usr
> 
> Prepare the symlinks required for merged /usr at the time of rootfs creation.
> 
> The links created in rootfs are:
> /bin --> /usr/sbin
> /sbin --> /usr/sbin
> /lib --> /usr/lib
> /lib64 --> /usr/lib64
> 
> We cannot make these symlinks as part of 'base-files' or some other package.
> Because at rootfs creation, installation of the package(say kernel) that depends
> on these root folders/links fails, if package manager installs this package
> prior to base-files.
> 
> These symbolic links in top level folder should present as long as

Change "should" to "should be".

>  - kerenl tools use /lib/{module,firmware}

Change "kerenl" to "kernel".

>  - shell scripts uses "#!/bin/sh"
> 
> Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
> ---
>  meta/classes/image.bbclass | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 85f6b9a..f08e3e1 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -617,3 +617,26 @@ do_bundle_initramfs () {
>  	:
>  }
>  addtask bundle_initramfs after do_image_complete
> +
> +# Prepare the root links to point to the /usr counterparts.

In the function below:
* You do not need to specify -m 0755 to install, it is the default.
* Change all "/${var}" to "${var}".

> +create_merged_usr_symlinks() {
> +    install -m 0755 -d ${IMAGE_ROOTFS}/${base_bindir}
> +    install -m 0755 -d ${IMAGE_ROOTFS}/${base_sbindir}
> +    install -m 0755 -d ${IMAGE_ROOTFS}/${base_libdir}

You can combine these three installs into one. Saves two forks.

> +    lnr ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}/bin
> +    lnr ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}/sbin
> +    lnr ${IMAGE_ROOTFS}${base_libdir} ${IMAGE_ROOTFS}/${baselib}
> +
> +    if [ "${nonarch_base_libdir}" != "${base_libdir}" ]; then

Why do you check "${nonarch_base_libdir}" != "${base_libdir}" when you 
do not check the other three above?

> +       install -m 0755 -d ${IMAGE_ROOTFS}/${nonarch_base_libdir}
> +       lnr ${IMAGE_ROOTFS}${nonarch_base_libdir} ${IMAGE_ROOTFS}/lib
> +    fi
> +
> +    # create base links for multilibs
> +    multi_libdirs="${@d.getVar('MULTILIB_VARIANTS')}"
> +    for d in $multi_libdirs; do
> +        install -m 0755 -d ${IMAGE_ROOTFS}/${exec_prefix}/$d
> +        lnr ${IMAGE_ROOTFS}/${exec_prefix}/$d ${IMAGE_ROOTFS}/$d
> +    done
> +}
> +ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks; ', '',d)}"
> --
> 2.7.4

//Peter

Re: [PATCH v3 7/8] image: create symlinks needed for merged /usr

[Valluri, Amarnath]

On Tue, 2017-06-13 at 08:05 +0000, Peter Kjellerstedt wrote:
> > 
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> > Of
> > Amarnath Valluri
> > Sent: den 13 juni 2017 08:53
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCH v3 7/8] image: create symlinks needed for
> > merged /usr
> > 
> > Prepare the symlinks required for merged /usr at the time of rootfs
> > creation.
> > 
> > The links created in rootfs are:
> > /bin --> /usr/sbin
> > /sbin --> /usr/sbin
> > /lib --> /usr/lib
> > /lib64 --> /usr/lib64
> > 
> > We cannot make these symlinks as part of 'base-files' or some other
> > package.
> > Because at rootfs creation, installation of the package(say kernel)
> > that depends
> > on these root folders/links fails, if package manager installs this
> > package
> > prior to base-files.
> > 
> > These symbolic links in top level folder should present as long as
> Change "should" to "should be".
> 
> > 
> >  - kerenl tools use /lib/{module,firmware}
> Change "kerenl" to "kernel".
> 
> > 
> >  - shell scripts uses "#!/bin/sh"
> > 
> > Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
> > ---
> >  meta/classes/image.bbclass | 23 +++++++++++++++++++++++
> >  1 file changed, 23 insertions(+)
> > 
> > diff --git a/meta/classes/image.bbclass
> > b/meta/classes/image.bbclass
> > index 85f6b9a..f08e3e1 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -617,3 +617,26 @@ do_bundle_initramfs () {
> >  	:
> >  }
> >  addtask bundle_initramfs after do_image_complete
> > +
> > +# Prepare the root links to point to the /usr counterparts.
> In the function below:
> * You do not need to specify -m 0755 to install, it is the default.
> * Change all "/${var}" to "${var}".
> 
> > 
> > +create_merged_usr_symlinks() {
> > +    install -m 0755 -d ${IMAGE_ROOTFS}/${base_bindir}
> > +    install -m 0755 -d ${IMAGE_ROOTFS}/${base_sbindir}
> > +    install -m 0755 -d ${IMAGE_ROOTFS}/${base_libdir}
> You can combine these three installs into one. Saves two forks.
> 
> > 
> > +    lnr ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}/bin
> > +    lnr ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}/sbin
> > +    lnr ${IMAGE_ROOTFS}${base_libdir} ${IMAGE_ROOTFS}/${baselib}
> > +
> > +    if [ "${nonarch_base_libdir}" != "${base_libdir}" ]; then
> Why do you check "${nonarch_base_libdir}" != "${base_libdir}" when
> you do not check the other three above?
Our intention here is to create /lib -> ${noarch_base_libdir}. This
might already been crated in the previous call : 
    lnr ${IMAGE_ROOTFS}${base_libdir} ${IMAGE_ROOTFS}/${baselib}

because, for non-multilib builds both ${nonarch_base_libdir} and
${base_libdir} points to ${exec_prefix}/lib.

The condition can be rephrased to:
   if [ "${baselib}" != "lib" ]

This case does not apply for other two links., ie., /bin, /sbin.

- Amarnath

[PATCH v4 7/8] image: create symlinks needed for merged /usr

[Amarnath Valluri]

Prepare the symlinks required for merged /usr at the time of rootfs creation.

The links created in rootfs are:
/bin --> /usr/sbin
/sbin --> /usr/sbin
/lib --> /usr/lib
/lib64 --> /usr/lib64

We cannot make these symlinks as part of 'base-files' or some other package.
Because at rootfs creation, installation of the package(say kernel) that depends
on these root folders/links fails, if package manager installs this package
prior to base-files.

These symbolic links in top level folder should be present as long as
 - kernel tools use /lib/{module,firmware}
 - shell scripts uses "#!/bin/sh"

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/classes/image.bbclass | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 85f6b9a..2a3c1e3 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -617,3 +617,24 @@ do_bundle_initramfs () {
 	:
 }
 addtask bundle_initramfs after do_image_complete
+
+# Prepare the root links to point to the /usr counterparts.
+create_merged_usr_symlinks() {
+    install -d ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}${base_libdir}
+    lnr ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}/bin
+    lnr ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}/sbin
+    lnr ${IMAGE_ROOTFS}${base_libdir} ${IMAGE_ROOTFS}/${baselib}
+
+    if [ "${nonarch_base_libdir}" != "${base_libdir}" ]; then
+       install -d ${IMAGE_ROOTFS}${nonarch_base_libdir}
+       lnr ${IMAGE_ROOTFS}${nonarch_base_libdir} ${IMAGE_ROOTFS}/lib
+    fi
+
+    # create base links for multilibs
+    multi_libdirs="${@d.getVar('MULTILIB_VARIANTS')}"
+    for d in $multi_libdirs; do
+        install -d ${IMAGE_ROOTFS}${exec_prefix}/$d
+        lnr ${IMAGE_ROOTFS}${exec_prefix}/$d ${IMAGE_ROOTFS}/$d
+    done
+}
+ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks; ', '',d)}"
-- 
2.7.4

Re: [PATCH v4 7/8] image: create symlinks needed for merged /usr

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Amarnath Valluri
> Sent: den 13 juni 2017 15:24
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH v4 7/8] image: create symlinks needed for
> merged /usr
> 
> Prepare the symlinks required for merged /usr at the time of rootfs
> creation.
> 
> The links created in rootfs are:
> /bin --> /usr/sbin
> /sbin --> /usr/sbin
> /lib --> /usr/lib
> /lib64 --> /usr/lib64
> 
> We cannot make these symlinks as part of 'base-files' or some other
> package.
> Because at rootfs creation, installation of the package(say kernel)
> that depends
> on these root folders/links fails, if package manager installs this
> package
> prior to base-files.
> 
> These symbolic links in top level folder should be present as long as
>  - kernel tools use /lib/{module,firmware}
>  - shell scripts uses "#!/bin/sh"
> 
> Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
> ---
>  meta/classes/image.bbclass | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 85f6b9a..2a3c1e3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -617,3 +617,24 @@ do_bundle_initramfs () {
>  	:
>  }
>  addtask bundle_initramfs after do_image_complete
> +
> +# Prepare the root links to point to the /usr counterparts.
> +create_merged_usr_symlinks() {
> +    install -d ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}${base_libdir}
> +    lnr ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}/bin
> +    lnr ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}/sbin
> +    lnr ${IMAGE_ROOTFS}${base_libdir} ${IMAGE_ROOTFS}/${baselib}

You missed one. Change "/${baselib}" to "${baselib}".

> +
> +    if [ "${nonarch_base_libdir}" != "${base_libdir}" ]; then
> +       install -d ${IMAGE_ROOTFS}${nonarch_base_libdir}
> +       lnr ${IMAGE_ROOTFS}${nonarch_base_libdir} ${IMAGE_ROOTFS}/lib
> +    fi
> +
> +    # create base links for multilibs
> +    multi_libdirs="${@d.getVar('MULTILIB_VARIANTS')}"
> +    for d in $multi_libdirs; do
> +        install -d ${IMAGE_ROOTFS}${exec_prefix}/$d
> +        lnr ${IMAGE_ROOTFS}${exec_prefix}/$d ${IMAGE_ROOTFS}/$d
> +    done
> +}
> +ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks; ', '',d)}"
> --
> 2.7.4

//Peter

Re: [PATCH v4 7/8] image: create symlinks needed for merged /usr

[Amarnath Valluri]

On Tue, 2017-06-13 at 14:16 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> > Amarnath Valluri
> > Sent: den 13 juni 2017 15:24
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCH v4 7/8] image: create symlinks needed for
> > merged /usr
> > 
> > Prepare the symlinks required for merged /usr at the time of rootfs
> > creation.
> > 
> > The links created in rootfs are:
> > /bin --> /usr/sbin
> > /sbin --> /usr/sbin
> > /lib --> /usr/lib
> > /lib64 --> /usr/lib64
> > 
> > We cannot make these symlinks as part of 'base-files' or some other
> > package.
> > Because at rootfs creation, installation of the package(say kernel)
> > that depends
> > on these root folders/links fails, if package manager installs this
> > package
> > prior to base-files.
> > 
> > These symbolic links in top level folder should be present as long as
> >  - kernel tools use /lib/{module,firmware}
> >  - shell scripts uses "#!/bin/sh"
> > 
> > Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
> > ---
> >  meta/classes/image.bbclass | 21 +++++++++++++++++++++
> >  1 file changed, 21 insertions(+)
> > 
> > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> > index 85f6b9a..2a3c1e3 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -617,3 +617,24 @@ do_bundle_initramfs () {
> >  	:
> >  }
> >  addtask bundle_initramfs after do_image_complete
> > +
> > +# Prepare the root links to point to the /usr counterparts.
> > +create_merged_usr_symlinks() {
> > +    install -d ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}${base_libdir}
> > +    lnr ${IMAGE_ROOTFS}${base_bindir} ${IMAGE_ROOTFS}/bin
> > +    lnr ${IMAGE_ROOTFS}${base_sbindir} ${IMAGE_ROOTFS}/sbin
> > +    lnr ${IMAGE_ROOTFS}${base_libdir} ${IMAGE_ROOTFS}/${baselib}
> 
> You missed one. Change "/${baselib}" to "${baselib}".
${baselib} is not directory path and does not prefix with '/' unlike
${base_libdir}, so its valid here.

- Amarnath

[PATCH v3 8/8] insane.bbclass: Add package QA check for merged /usr.

[Amarnath Valluri]

This check makes sure that, when usrmerge distro feature enabled, no package
installs files to root (/bin, /sbin, /lib, /lib64) folders.

Signed-off-by: Amarnath Valluri <amarnath.valluri@intel.com>
---
 meta/classes/insane.bbclass | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 5a3d017..1ee7f6b 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -40,6 +40,9 @@ ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             version-going-backwards expanded-d invalid-chars \
             license-checksum dev-elf file-rdeps \
             "
+# Add usrmerge QA check based on distro feature	    
+ERROR_QA_append = "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)}"
+
 FAKEROOT_QA = "host-user-contaminated"
 FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \
 enabled tests are listed here, the do_package_qa task will run under fakeroot."
@@ -993,6 +996,18 @@ def package_qa_check_deps(pkg, pkgdest, skip, d):
     check_valid_deps('RREPLACES')
     check_valid_deps('RCONFLICTS')
 
+QAPKGTEST[usrmerge] = "package_qa_check_usrmerge"
+def package_qa_check_usrmerge(pkg, d, messages):
+    pkgdest = d.getVar('PKGDEST')
+    pkg_dir = pkgdest + os.sep + pkg + os.sep
+    merged_dirs = ['bin', 'sbin', 'lib'] + d.getVar('MULTILIB_VARIANTS').split()
+    for f in merged_dirs:
+        if os.path.exists(pkg_dir + f) and not os.path.islink(pkg_dir + f):
+            msg = "%s package is not obeying usrmerge distro feature. /%s should be relocated to /usr." % (pkg, f)
+            package_qa_add_message(messages, "usrmerge", msg)
+            return False
+    return True
+
 QAPKGTEST[expanded-d] = "package_qa_check_expanded_d"
 def package_qa_check_expanded_d(package, d, messages):
     """
@@ -1066,6 +1081,7 @@ def package_qa_check_host_user(path, name, d, elf, messages):
             return False
     return True
 
+
 # The PACKAGE FUNC to scan each package
 python do_package_qa () {
     import subprocess
-- 
2.7.4