* [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
@ 2019-02-07 0:35 Khem Raj
2019-02-07 9:00 ` Richard Purdie
` (2 more replies)
0 siblings, 3 replies; 22+ messages in thread
From: Khem Raj @ 2019-02-07 0:35 UTC (permalink / raw)
To: openembedded-core
This has been a constant source of trouble for build failures due to host-user-contaminated QA
errors of sort
ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid 3004, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
So far we have tried to mould cp command into not carrying the build
user permissions into install area but it is never entirely fixed since
the issue keeps popping up in various scenes
This patch replaces use of cp with install utility and specifies install
mode for files explcitly
Signed-off-by: Khem Raj <raj.khem@gmail.com>
---
meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
1 file changed, 25 insertions(+), 19 deletions(-)
diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
index 6384f9cbf1..9b256a5108 100644
--- a/meta/recipes-core/glibc/glibc-locale.inc
+++ b/meta/recipes-core/glibc/glibc-locale.inc
@@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
do_install () {
- mkdir -p ${D}${bindir} ${D}${datadir}
- if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
- fi
- if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
- mkdir -p ${D}${localedir}
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
- fi
+ install -d ${D}${bindir}
+ find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
+ -exec install -m 0755 -t "${D}${bindir}" {} \;
+
+ for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -printf '%P ') ; do
+ install -d "${D}${localedir}/$d"
+ find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1 -type f \
+ -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
+ done
if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
- mkdir -p ${D}${libdir}
- if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
- fi
- if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
- fi
- fi
- if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
+ for d in . $(find "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
+ install -d "${D}${libdir}/gconv/$d"
+ find "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
+ -exec install -m 0755 -t "${D}${libdir}/gconv/$d" {} \;
+ done
+ for d in . $(find "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
+ install -d "${D}${datadir}/i18n/$d"
+ find "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
+ -exec install -m 0644 -t "${D}${datadir}/i18n/$d" {} \;
+ done
fi
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
+ for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type d -printf '%P ') ; do
+ install -d "${D}${datadir}/locale/$d"
+ find "${LOCALETREESRC}/${datadir}/locale/$d" -maxdepth 1 -type f \
+ -exec install -m 0644 -t "${D}${datadir}/locale/$d" {} \;
+ done
+ install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
}
inherit libc-package
--
2.20.1
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 0:35 [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp Khem Raj
@ 2019-02-07 9:00 ` Richard Purdie
2019-02-07 14:49 ` Khem Raj
2019-02-10 1:29 ` Peter Kjellerstedt
2019-02-07 11:44 ` Peter Kjellerstedt
2019-02-28 1:53 ` ChenQi
2 siblings, 2 replies; 22+ messages in thread
From: Richard Purdie @ 2019-02-07 9:00 UTC (permalink / raw)
To: Khem Raj, openembedded-core
On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> This has been a constant source of trouble for build failures due to host-user-contaminated QA
> errors of sort
>
> ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid 3004, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
>
> So far we have tried to mould cp command into not carrying the build
> user permissions into install area but it is never entirely fixed since
> the issue keeps popping up in various scenes
>
> This patch replaces use of cp with install utility and specifies install
> mode for files explcitly
>
> Signed-off-by: Khem Raj <raj.khem@gmail.com>
> ---
> meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
> 1 file changed, 25 insertions(+), 19 deletions(-)
>
> diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> index 6384f9cbf1..9b256a5108 100644
> --- a/meta/recipes-core/glibc/glibc-locale.inc
> +++ b/meta/recipes-core/glibc/glibc-locale.inc
> @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
>
> do_install () {
> - mkdir -p ${D}${bindir} ${D}${datadir}
> - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> - fi
> - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> - mkdir -p ${D}${localedir}
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> - fi
> + install -d ${D}${bindir}
> + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> + -exec install -m 0755 -t "${D}${bindir}" {} \;
> +
> + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -printf '%P ') ; do
> + install -d "${D}${localedir}/$d"
> + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> + done
> if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> - mkdir -p ${D}${libdir}
> - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> - fi
> - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> - fi
> - fi
> - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
> + install -d "${D}${libdir}/gconv/$d"
> + find "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
> + -exec install -m 0755 -t "${D}${libdir}/gconv/$d" {} \;
> + done
> + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
> + install -d "${D}${datadir}/i18n/$d"
> + find "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${datadir}/i18n/$d" {} \;
> + done
> fi
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type d -printf '%P ') ; do
> + install -d "${D}${datadir}/locale/$d"
> + find "${LOCALETREESRC}/${datadir}/locale/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${datadir}/locale/$d" {} \;
> + done
> + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> }
>
> inherit libc-package
The trouble is this is a workaround. The cp commands should work and
there is some underlying issue in pseudo causing this. We really need
to figure out what that is since this will likely just mean we see the
problem somewhere else :(
I still suspect some kind of inode number reuse problem which these cp
commands trigger...
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 0:35 [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp Khem Raj
2019-02-07 9:00 ` Richard Purdie
@ 2019-02-07 11:44 ` Peter Kjellerstedt
2019-02-07 14:53 ` Khem Raj
2019-02-28 1:53 ` ChenQi
2 siblings, 1 reply; 22+ messages in thread
From: Peter Kjellerstedt @ 2019-02-07 11:44 UTC (permalink / raw)
To: Khem Raj, openembedded-core@lists.openembedded.org
> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Khem Raj
> Sent: den 7 februari 2019 01:36
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> install utility instead of cp
>
> This has been a constant source of trouble for build failures due to
> host-user-contaminated QA errors of sort
>
> ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid
> 3004, which is the same as the user running bitbake. This may be due to
> host contamination [host-user-contaminated]
>
> So far we have tried to mould cp command into not carrying the build
> user permissions into install area but it is never entirely fixed since
> the issue keeps popping up in various scenes
>
> This patch replaces use of cp with install utility and specifies
> install mode for files explcitly
>
> Signed-off-by: Khem Raj <raj.khem@gmail.com>
> ---
> meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
> 1 file changed, 25 insertions(+), 19 deletions(-)
>
> diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> index 6384f9cbf1..9b256a5108 100644
> --- a/meta/recipes-core/glibc/glibc-locale.inc
> +++ b/meta/recipes-core/glibc/glibc-locale.inc
> @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
>
> do_install () {
> - mkdir -p ${D}${bindir} ${D}${datadir}
> - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> - fi
> - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> - mkdir -p ${D}${localedir}
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> - fi
> + install -d ${D}${bindir}
> + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> + -exec install -m 0755 -t "${D}${bindir}" {} \;
> +
> + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -printf '%P ') ; do
> + install -d "${D}${localedir}/$d"
> + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> + done
> if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> - mkdir -p ${D}${libdir}
> - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> - fi
> - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> - fi
> - fi
> - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
> + install -d "${D}${libdir}/gconv/$d"
> + find "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
> + -exec install -m 0755 -t "${D}${libdir}/gconv/$d" {} \;
> + done
> + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
> + install -d "${D}${datadir}/i18n/$d"
> + find "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${datadir}/i18n/$d" {} \;
> + done
> fi
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type d -printf '%P ') ; do
> + install -d "${D}${datadir}/locale/$d"
> + find "${LOCALETREESRC}/${datadir}/locale/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${datadir}/locale/$d" {} \;
> + done
> + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> }
>
> inherit libc-package
> --
> 2.20.1
May I suggest using a help function to simplify the code a lot:
copy_locale_files() {
local dir=$1 mode=$2
[ -e "${LOCALETREESRC}$dir" ] || return 0
for d in . $(find "${LOCALETREESRC}$dir" -type d -printf '%P '); do
install -d ${D}$dir/$d
find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
-exec install -m $mode -t "${D}$dir/$d" {} \;
done
}
do_install() {
copy_locale_files ${bindir} 0755
copy_locale_files ${localedir} 0644
if [ ${PACKAGE_NO_GCONV} -eq 0 ]; then
copy_locale_files ${libdir}/gconv 0755
copy_locale_files ${datadir}/i18n 0644
fi
copy_locale_files ${datadir}/locale 0644
install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
}
The code above is untested, but I think I got it reasonably correct.
Also note that I have restored the indentation to use tabs, and
turned the access to the ${PACKAGE_NO_GCONV} variable into a
simple variable instead of using d.getVar().
//Peter
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 9:00 ` Richard Purdie
@ 2019-02-07 14:49 ` Khem Raj
2019-02-07 16:17 ` Martin Jansa
2019-02-10 1:29 ` Peter Kjellerstedt
1 sibling, 1 reply; 22+ messages in thread
From: Khem Raj @ 2019-02-07 14:49 UTC (permalink / raw)
To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer
On Thu, Feb 7, 2019 at 1:00 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > This has been a constant source of trouble for build failures due to host-user-contaminated QA
> > errors of sort
> >
> > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid 3004, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
> >
> > So far we have tried to mould cp command into not carrying the build
> > user permissions into install area but it is never entirely fixed since
> > the issue keeps popping up in various scenes
> >
> > This patch replaces use of cp with install utility and specifies install
> > mode for files explcitly
> >
> > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > ---
> > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
> > 1 file changed, 25 insertions(+), 19 deletions(-)
> >
> > diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> > index 6384f9cbf1..9b256a5108 100644
> > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
> >
> > do_install () {
> > - mkdir -p ${D}${bindir} ${D}${datadir}
> > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > - fi
> > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > - mkdir -p ${D}${localedir}
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > - fi
> > + install -d ${D}${bindir}
> > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > +
> > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -printf '%P ') ; do
> > + install -d "${D}${localedir}/$d"
> > + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1 -type f \
> > + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> > + done
> > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > - mkdir -p ${D}${libdir}
> > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > - fi
> > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > - fi
> > - fi
> > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
> > + install -d "${D}${libdir}/gconv/$d"
> > + find "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
> > + -exec install -m 0755 -t "${D}${libdir}/gconv/$d" {} \;
> > + done
> > + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
> > + install -d "${D}${datadir}/i18n/$d"
> > + find "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
> > + -exec install -m 0644 -t "${D}${datadir}/i18n/$d" {} \;
> > + done
> > fi
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type d -printf '%P ') ; do
> > + install -d "${D}${datadir}/locale/$d"
> > + find "${LOCALETREESRC}/${datadir}/locale/$d" -maxdepth 1 -type f \
> > + -exec install -m 0644 -t "${D}${datadir}/locale/$d" {} \;
> > + done
> > + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> > }
> >
> > inherit libc-package
>
> The trouble is this is a workaround. The cp commands should work and
> there is some underlying issue in pseudo causing this. We really need
> to figure out what that is since this will likely just mean we see the
> problem somewhere else :(
so far this is the only place where it shows in world builds that
meta-oe does which includes many layers.
>
> I still suspect some kind of inode number reuse problem which these cp
> commands trigger...
cp and install are not same when it comes to how they operate
underneath. install is better when it comes to what we are doing here
it also gives better control of permissions and mods,as a recipe writer
someone would prefer then to let the tool ( cp ) assume it.
So there might be a pseudo issue, I dont disagree but not obvious and
I believe using cp is subpar anyway.
>
> Cheers,
>
> Richard
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 11:44 ` Peter Kjellerstedt
@ 2019-02-07 14:53 ` Khem Raj
0 siblings, 0 replies; 22+ messages in thread
From: Khem Raj @ 2019-02-07 14:53 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: openembedded-core@lists.openembedded.org
On Thu, Feb 7, 2019 at 3:44 AM Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> > core-bounces@lists.openembedded.org> On Behalf Of Khem Raj
> > Sent: den 7 februari 2019 01:36
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > install utility instead of cp
> >
> > This has been a constant source of trouble for build failures due to
> > host-user-contaminated QA errors of sort
> >
> > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> > es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid
> > 3004, which is the same as the user running bitbake. This may be due to
> > host contamination [host-user-contaminated]
> >
> > So far we have tried to mould cp command into not carrying the build
> > user permissions into install area but it is never entirely fixed since
> > the issue keeps popping up in various scenes
> >
> > This patch replaces use of cp with install utility and specifies
> > install mode for files explcitly
> >
> > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > ---
> > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
> > 1 file changed, 25 insertions(+), 19 deletions(-)
> >
> > diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> > index 6384f9cbf1..9b256a5108 100644
> > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
> >
> > do_install () {
> > - mkdir -p ${D}${bindir} ${D}${datadir}
> > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > - fi
> > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > - mkdir -p ${D}${localedir}
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > - fi
> > + install -d ${D}${bindir}
> > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > +
> > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -printf '%P ') ; do
> > + install -d "${D}${localedir}/$d"
> > + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1 -type f \
> > + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> > + done
> > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > - mkdir -p ${D}${libdir}
> > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > - fi
> > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > - fi
> > - fi
> > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
> > + install -d "${D}${libdir}/gconv/$d"
> > + find "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
> > + -exec install -m 0755 -t "${D}${libdir}/gconv/$d" {} \;
> > + done
> > + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
> > + install -d "${D}${datadir}/i18n/$d"
> > + find "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
> > + -exec install -m 0644 -t "${D}${datadir}/i18n/$d" {} \;
> > + done
> > fi
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type d -printf '%P ') ; do
> > + install -d "${D}${datadir}/locale/$d"
> > + find "${LOCALETREESRC}/${datadir}/locale/$d" -maxdepth 1 -type f \
> > + -exec install -m 0644 -t "${D}${datadir}/locale/$d" {} \;
> > + done
> > + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> > }
> >
> > inherit libc-package
> > --
> > 2.20.1
>
> May I suggest using a help function to simplify the code a lot:
>
> copy_locale_files() {
> local dir=$1 mode=$2
>
> [ -e "${LOCALETREESRC}$dir" ] || return 0
>
> for d in . $(find "${LOCALETREESRC}$dir" -type d -printf '%P '); do
> install -d ${D}$dir/$d
> find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
> -exec install -m $mode -t "${D}$dir/$d" {} \;
> done
> }
>
> do_install() {
> copy_locale_files ${bindir} 0755
> copy_locale_files ${localedir} 0644
> if [ ${PACKAGE_NO_GCONV} -eq 0 ]; then
> copy_locale_files ${libdir}/gconv 0755
> copy_locale_files ${datadir}/i18n 0644
> fi
> copy_locale_files ${datadir}/locale 0644
> install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> }
>
> The code above is untested, but I think I got it reasonably correct.
> Also note that I have restored the indentation to use tabs, and
> turned the access to the ${PACKAGE_NO_GCONV} variable into a
> simple variable instead of using d.getVar().
>
thats a neat idea. I will send a v2 with this
> //Peter
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 14:49 ` Khem Raj
@ 2019-02-07 16:17 ` Martin Jansa
2019-02-07 16:44 ` Joshua Watt
0 siblings, 1 reply; 22+ messages in thread
From: Martin Jansa @ 2019-02-07 16:17 UTC (permalink / raw)
To: Khem Raj; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 6771 bytes --]
There are other rare reproducers which don't involve cp/install (e.g. older
qmake was triggering it quite often as well), I've mentioned some in:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434
glibc-locale is the most common one for sure as pretty much everybody is
building that one, I don't have objection converting this from cp to
install to prevent this random error (very rare random failure doesn't
really help with #12434 reproducer and whoever work on it, can either
revert this change or try to reproduce with something else) - I've included
few hundreds of INSANE_SKIPs to ignore _random_ host-user-contaminated QAs
in LG layers to prevent our builds failing randomly and the one for
glibc-locale is the longest part:
https://github.com/webosose/meta-webosose/blob/master/meta-webos/recipes-core/glibc/glibc-locale_%25.bbappend
Regards,
On Thu, Feb 7, 2019 at 3:50 PM Khem Raj <raj.khem@gmail.com> wrote:
> On Thu, Feb 7, 2019 at 1:00 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > > This has been a constant source of trouble for build failures due to
> host-user-contaminated QA
> > > errors of sort
> > >
> > > ERROR: QA Issue: glibc-locale:
> /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY
> is owned by uid 3004, which is the same as the user running bitbake. This
> may be due to host contamination [host-user-contaminated]
> > >
> > > So far we have tried to mould cp command into not carrying the build
> > > user permissions into install area but it is never entirely fixed since
> > > the issue keeps popping up in various scenes
> > >
> > > This patch replaces use of cp with install utility and specifies
> install
> > > mode for files explcitly
> > >
> > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > ---
> > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
> > > 1 file changed, 25 insertions(+), 19 deletions(-)
> > >
> > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc
> b/meta/recipes-core/glibc/glibc-locale.inc
> > > index 6384f9cbf1..9b256a5108 100644
> > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
> > >
> > > do_install () {
> > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > - fi
> > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > - mkdir -p ${D}${localedir}
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > - fi
> > > + install -d ${D}${bindir}
> > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > > +
> > > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d
> -printf '%P ') ; do
> > > + install -d "${D}${localedir}/$d"
> > > + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1
> -type f \
> > > + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> > > + done
> > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > - mkdir -p ${D}${libdir}
> > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > - fi
> > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > - fi
> > > - fi
> > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv"
> -type d -printf '%P ') ; do
> > > + install -d "${D}${libdir}/gconv/$d"
> > > + find "${LOCALETREESRC}/${libdir}/gconv/$d"
> -maxdepth 1 -type f \
> > > + -exec install -m 0755 -t
> "${D}${libdir}/gconv/$d" {} \;
> > > + done
> > > + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n"
> -type d -printf '%P ') ; do
> > > + install -d "${D}${datadir}/i18n/$d"
> > > + find "${LOCALETREESRC}/${datadir}/i18n/$d"
> -maxdepth 1 -type f \
> > > + -exec install -m 0644 -t
> "${D}${datadir}/i18n/$d" {} \;
> > > + done
> > > fi
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type
> d -printf '%P ') ; do
> > > + install -d "${D}${datadir}/locale/$d"
> > > + find "${LOCALETREESRC}/${datadir}/locale/$d"
> -maxdepth 1 -type f \
> > > + -exec install -m 0644 -t "${D}${datadir}/locale/$d"
> {} \;
> > > + done
> > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> > > }
> > >
> > > inherit libc-package
> >
> > The trouble is this is a workaround. The cp commands should work and
> > there is some underlying issue in pseudo causing this. We really need
> > to figure out what that is since this will likely just mean we see the
> > problem somewhere else :(
>
> so far this is the only place where it shows in world builds that
> meta-oe does which includes many layers.
>
> >
> > I still suspect some kind of inode number reuse problem which these cp
> > commands trigger...
>
> cp and install are not same when it comes to how they operate
> underneath. install is better when it comes to what we are doing here
> it also gives better control of permissions and mods,as a recipe writer
> someone would prefer then to let the tool ( cp ) assume it.
>
> So there might be a pseudo issue, I dont disagree but not obvious and
> I believe using cp is subpar anyway.
>
> >
> > Cheers,
> >
> > Richard
> >
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
[-- Attachment #2: Type: text/html, Size: 8900 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 16:17 ` Martin Jansa
@ 2019-02-07 16:44 ` Joshua Watt
2019-02-07 16:59 ` Joshua Watt
0 siblings, 1 reply; 22+ messages in thread
From: Joshua Watt @ 2019-02-07 16:44 UTC (permalink / raw)
To: Martin Jansa, Khem Raj; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 9065 bytes --]
Martin,
Do you by any chance build in Docker (or some other transient
container)? We recently found an issue with building in docker where
when the primary container command would exit, causing the container to
terminate. When the container terminates, the kernel unceremoniously
sends SIGKILL to all its processes. This plays *very* poorly with
pseudo because pseudo keeps it's entire database in a in-memory sqlite
database and only flushes it out to disk on a periodic interval, or
when it get SIGTERM (maybe SIGQUIT?) or exits normally. By default the
pseudo daemon will also hang around for 20(?) seconds waiting for a new
connection from a pseudo client, meaning that when our docker container
was exiting, there were potentially several pseudo dameons sitting
around waiting for a client with unflushed databases in memory that
suddenly got a SIGKILL and all their data was lost. This was
particularly bad if you aborted a build while it was running, or some
build failure occurred. Since the pseudo database was lost, the only
way to correct these errors was to clean and rebuild.
I think pseudo could handle this better and keep the database on disk
instead of in memory, but in the short term you can hack bitbake to
pass the -S flag when invoking the pseudo client which will make it
tell the dameon to shutdown (and thus flush the database) if it is the
last connection.
On Thu, 2019-02-07 at 17:17 +0100, Martin Jansa wrote:
> There are other rare reproducers which don't involve cp/install (e.g.
> older qmake was triggering it quite often as well), I've mentioned
> some in:https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434
>
> glibc-locale is the most common one for sure as pretty much everybody
> is building that one, I don't have objection converting this from cp
> to install to prevent this random error (very rare random failure
> doesn't really help with #12434 reproducer and whoever work on it,
> can either revert this change or try to reproduce with something
> else) - I've included few hundreds of INSANE_SKIPs to ignore _random_
> host-user-contaminated QAs in LG layers to prevent our builds failing
> randomly and the one for glibc-locale is the longest part:
> https://github.com/webosose/meta-webosose/blob/master/meta-webos/recipes-core/glibc/glibc-locale_%25.bbappend
>
> Regards,
>
> On Thu, Feb 7, 2019 at 3:50 PM Khem Raj <raj.khem@gmail.com> wrote:
> > On Thu, Feb 7, 2019 at 1:00 AM Richard Purdie
> >
> > <richard.purdie@linuxfoundation.org> wrote:
> >
> > >
> >
> > > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> >
> > > > This has been a constant source of trouble for build failures
> > due to host-user-contaminated QA
> >
> > > > errors of sort
> >
> > > >
> >
> > > > ERROR: QA Issue: glibc-locale:
> > /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia
> > /LC_MONETARY is owned by uid 3004, which is the same as the user
> > running bitbake. This may be due to host contamination [host-user-
> > contaminated]
> >
> > > >
> >
> > > > So far we have tried to mould cp command into not carrying the
> > build
> >
> > > > user permissions into install area but it is never entirely
> > fixed since
> >
> > > > the issue keeps popping up in various scenes
> >
> > > >
> >
> > > > This patch replaces use of cp with install utility and
> > specifies install
> >
> > > > mode for files explcitly
> >
> > > >
> >
> > > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> >
> > > > ---
> >
> > > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++
> > ----------
> >
> > > > 1 file changed, 25 insertions(+), 19 deletions(-)
> >
> > > >
> >
> > > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc
> > b/meta/recipes-core/glibc/glibc-locale.inc
> >
> > > > index 6384f9cbf1..9b256a5108 100644
> >
> > > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> >
> > > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> >
> > > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> >
> > > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-
> > stash-locale"
> >
> > > >
> >
> > > > do_install () {
> >
> > > > - mkdir -p ${D}${bindir} ${D}${datadir}
> >
> > > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> >
> > > > - cp -R --no-dereference --preserve=mode,links
> > ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> >
> > > > - fi
> >
> > > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> >
> > > > - mkdir -p ${D}${localedir}
> >
> > > > - cp -R --no-dereference --preserve=mode,links
> > ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> >
> > > > - fi
> >
> > > > + install -d ${D}${bindir}
> >
> > > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f
> > \
> >
> > > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> >
> > > > +
> >
> > > > + for d in . $(find "${LOCALETREESRC}/${localedir}"
> > -type d -printf '%P ') ; do
> >
> > > > + install -d "${D}${localedir}/$d"
> >
> > > > + find "${LOCALETREESRC}/${localedir}/$d"
> > -maxdepth 1 -type f \
> >
> > > > + -exec install -m 0644 -t "${D}${localedir}/$d"
> > {} \;
> >
> > > > + done
> >
> > > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> >
> > > > - mkdir -p ${D}${libdir}
> >
> > > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> >
> > > > - cp -R --no-dereference --
> > preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> >
> > > > - fi
> >
> > > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> >
> > > > - cp -R --no-dereference --
> > preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> >
> > > > - fi
> >
> > > > - fi
> >
> > > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> >
> > > > - cp -R --no-dereference --preserve=mode,links
> > ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> >
> > > > + for d in . $(find
> > "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
> >
> > > > + install -d "${D}${libdir}/gconv/$d"
> >
> > > > + find
> > "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
> >
> > > > + -exec install -m 0755 -t
> > "${D}${libdir}/gconv/$d" {} \;
> >
> > > > + done
> >
> > > > + for d in . $(find
> > "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
> >
> > > > + install -d "${D}${datadir}/i18n/$d"
> >
> > > > + find
> > "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
> >
> > > > + -exec install -m 0644 -t
> > "${D}${datadir}/i18n/$d" {} \;
> >
> > > > + done
> >
> > > > fi
> >
> > > > - cp -R --no-dereference --preserve=mode,links
> > ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> >
> > > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale"
> > -type d -printf '%P ') ; do
> >
> > > > + install -d "${D}${datadir}/locale/$d"
> >
> > > > + find "${LOCALETREESRC}/${datadir}/locale/$d"
> > -maxdepth 1 -type f \
> >
> > > > + -exec install -m 0644 -t
> > "${D}${datadir}/locale/$d" {} \;
> >
> > > > + done
> >
> > > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED
> > ${WORKDIR}/SUPPORTED
> >
> > > > }
> >
> > > >
> >
> > > > inherit libc-package
> >
> > >
> >
> > > The trouble is this is a workaround. The cp commands should work
> > and
> >
> > > there is some underlying issue in pseudo causing this. We really
> > need
> >
> > > to figure out what that is since this will likely just mean we
> > see the
> >
> > > problem somewhere else :(
> >
> >
> >
> > so far this is the only place where it shows in world builds that
> >
> > meta-oe does which includes many layers.
> >
> >
> >
> > >
> >
> > > I still suspect some kind of inode number reuse problem which
> > these cp
> >
> > > commands trigger...
> >
> >
> >
> > cp and install are not same when it comes to how they operate
> >
> > underneath. install is better when it comes to what we are doing
> > here
> >
> > it also gives better control of permissions and mods,as a recipe
> > writer
> >
> > someone would prefer then to let the tool ( cp ) assume it.
> >
> >
> >
> > So there might be a pseudo issue, I dont disagree but not obvious
> > and
> >
> > I believe using cp is subpar anyway.
> >
> >
> >
> > >
> >
> > > Cheers,
> >
> > >
> >
> > > Richard
> >
> > >
> >
--
Joshua Watt <JPEWhacker@gmail.com>
[-- Attachment #2: Type: text/html, Size: 11279 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 16:44 ` Joshua Watt
@ 2019-02-07 16:59 ` Joshua Watt
2019-02-07 17:11 ` Martin Jansa
2019-02-07 17:21 ` Richard Purdie
0 siblings, 2 replies; 22+ messages in thread
From: Joshua Watt @ 2019-02-07 16:59 UTC (permalink / raw)
To: Martin Jansa, Khem Raj; +Cc: Patches and discussions about the oe-core layer
On Thu, Feb 7, 2019 at 10:44 AM Joshua Watt <jpewhacker@gmail.com> wrote:
>
> Martin,
>
> Do you by any chance build in Docker (or some other transient container)? We recently found an issue with building in docker where when the primary container command would exit, causing the container to terminate. When the container terminates, the kernel unceremoniously sends SIGKILL to all its processes. This plays *very* poorly with pseudo because pseudo keeps it's entire database in a in-memory sqlite database and only flushes it out to disk on a periodic interval, or when it get SIGTERM (maybe SIGQUIT?) or exits normally. By default the pseudo daemon will also hang around for 20(?) seconds waiting for a new connection from a pseudo client, meaning that when our docker container was exiting, there were potentially several pseudo dameons sitting around waiting for a client with unflushed databases in memory that suddenly got a SIGKILL and all their data was lost. This was particularly bad if you aborted a build while it was running, or some build failure occurred. Since the pseudo database was lost, the only way to correct these errors was to clean and rebuild.
>
> I think pseudo could handle this better and keep the database on disk instead of in memory, but in the short term you can hack bitbake to pass the -S flag when invoking the pseudo client which will make it tell the dameon to shutdown (and thus flush the database) if it is the last connection.
FWIW, this patch makes it much easier to do that
http://lists.openembedded.org/pipermail/bitbake-devel/2019-February/019813.html
Just add
FAKEROOTCMD += "-S"
in local.conf
>
>
> On Thu, 2019-02-07 at 17:17 +0100, Martin Jansa wrote:
>
> There are other rare reproducers which don't involve cp/install (e.g. older qmake was triggering it quite often as well), I've mentioned some in:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434
>
> glibc-locale is the most common one for sure as pretty much everybody is building that one, I don't have objection converting this from cp to install to prevent this random error (very rare random failure doesn't really help with #12434 reproducer and whoever work on it, can either revert this change or try to reproduce with something else) - I've included few hundreds of INSANE_SKIPs to ignore _random_ host-user-contaminated QAs in LG layers to prevent our builds failing randomly and the one for glibc-locale is the longest part:
> https://github.com/webosose/meta-webosose/blob/master/meta-webos/recipes-core/glibc/glibc-locale_%25.bbappend
>
> Regards,
>
> On Thu, Feb 7, 2019 at 3:50 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Thu, Feb 7, 2019 at 1:00 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > > This has been a constant source of trouble for build failures due to host-user-contaminated QA
> > > errors of sort
> > >
> > > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid 3004, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
> > >
> > > So far we have tried to mould cp command into not carrying the build
> > > user permissions into install area but it is never entirely fixed since
> > > the issue keeps popping up in various scenes
> > >
> > > This patch replaces use of cp with install utility and specifies install
> > > mode for files explcitly
> > >
> > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > ---
> > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
> > > 1 file changed, 25 insertions(+), 19 deletions(-)
> > >
> > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> > > index 6384f9cbf1..9b256a5108 100644
> > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
> > >
> > > do_install () {
> > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > - fi
> > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > - mkdir -p ${D}${localedir}
> > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > - fi
> > > + install -d ${D}${bindir}
> > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > > +
> > > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -printf '%P ') ; do
> > > + install -d "${D}${localedir}/$d"
> > > + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1 -type f \
> > > + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> > > + done
> > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > - mkdir -p ${D}${libdir}
> > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > - fi
> > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > - fi
> > > - fi
> > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
> > > + install -d "${D}${libdir}/gconv/$d"
> > > + find "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
> > > + -exec install -m 0755 -t "${D}${libdir}/gconv/$d" {} \;
> > > + done
> > > + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
> > > + install -d "${D}${datadir}/i18n/$d"
> > > + find "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
> > > + -exec install -m 0644 -t "${D}${datadir}/i18n/$d" {} \;
> > > + done
> > > fi
> > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type d -printf '%P ') ; do
> > > + install -d "${D}${datadir}/locale/$d"
> > > + find "${LOCALETREESRC}/${datadir}/locale/$d" -maxdepth 1 -type f \
> > > + -exec install -m 0644 -t "${D}${datadir}/locale/$d" {} \;
> > > + done
> > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> > > }
> > >
> > > inherit libc-package
> >
> > The trouble is this is a workaround. The cp commands should work and
> > there is some underlying issue in pseudo causing this. We really need
> > to figure out what that is since this will likely just mean we see the
> > problem somewhere else :(
>
> so far this is the only place where it shows in world builds that
> meta-oe does which includes many layers.
>
> >
> > I still suspect some kind of inode number reuse problem which these cp
> > commands trigger...
>
> cp and install are not same when it comes to how they operate
> underneath. install is better when it comes to what we are doing here
> it also gives better control of permissions and mods,as a recipe writer
> someone would prefer then to let the tool ( cp ) assume it.
>
> So there might be a pseudo issue, I dont disagree but not obvious and
> I believe using cp is subpar anyway.
>
> >
> > Cheers,
> >
> > Richard
> >
>
> --
>
> Joshua Watt <JPEWhacker@gmail.com>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 16:59 ` Joshua Watt
@ 2019-02-07 17:11 ` Martin Jansa
2019-02-07 17:21 ` Richard Purdie
1 sibling, 0 replies; 22+ messages in thread
From: Martin Jansa @ 2019-02-07 17:11 UTC (permalink / raw)
To: Joshua Watt; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 9079 bytes --]
On Thu, Feb 07, 2019 at 10:59:18AM -0600, Joshua Watt wrote:
> On Thu, Feb 7, 2019 at 10:44 AM Joshua Watt <jpewhacker@gmail.com> wrote:
> >
> > Martin,
> >
> > Do you by any chance build in Docker (or some other transient container)? We recently found an issue with building in docker where when the primary container command would exit, causing the container to terminate. When the container terminates, the kernel unceremoniously sends SIGKILL to all its processes. This plays *very* poorly with pseudo because pseudo keeps it's entire database in a in-memory sqlite database and only flushes it out to disk on a periodic interval, or when it get SIGTERM (maybe SIGQUIT?) or exits normally. By default the pseudo daemon will also hang around for 20(?) seconds waiting for a new connection from a pseudo client, meaning that when our docker container was exiting, there were potentially several pseudo dameons sitting around waiting for a client with unflushed databases in memory that suddenly got a SIGKILL and all their data was lost. This was particularly bad if you aborted a build while it was running, or some build failure occurred. Since the pseudo database was lost, the only way to correct these errors was to clean and rebuild.
> >
> > I think pseudo could handle this better and keep the database on disk instead of in memory, but in the short term you can hack bitbake to pass the -S flag when invoking the pseudo client which will make it tell the dameon to shutdown (and thus flush the database) if it is the last connection.
>
> FWIW, this patch makes it much easier to do that
> http://lists.openembedded.org/pipermail/bitbake-devel/2019-February/019813.html
> Just add
> FAKEROOTCMD += "-S"
> in local.conf
Hi Joshua,
most of the build failures caused by this weren't from the builds in
Docker (or any other container), just plain Ubuntu (various versions
14.04 - 18.04) with just the prerequisites installed.
When I get access to some bigger build farm (probably in few months) I
can run that for loop reproducer from the ticket again with
FAKEROOTCMD += "-S"
to see if it makes any difference.
Regards,
> > On Thu, 2019-02-07 at 17:17 +0100, Martin Jansa wrote:
> >
> > There are other rare reproducers which don't involve cp/install (e.g. older qmake was triggering it quite often as well), I've mentioned some in:
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434
> >
> > glibc-locale is the most common one for sure as pretty much everybody is building that one, I don't have objection converting this from cp to install to prevent this random error (very rare random failure doesn't really help with #12434 reproducer and whoever work on it, can either revert this change or try to reproduce with something else) - I've included few hundreds of INSANE_SKIPs to ignore _random_ host-user-contaminated QAs in LG layers to prevent our builds failing randomly and the one for glibc-locale is the longest part:
> > https://github.com/webosose/meta-webosose/blob/master/meta-webos/recipes-core/glibc/glibc-locale_%25.bbappend
> >
> > Regards,
> >
> > On Thu, Feb 7, 2019 at 3:50 PM Khem Raj <raj.khem@gmail.com> wrote:
> >
> > On Thu, Feb 7, 2019 at 1:00 AM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > >
> > > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > > > This has been a constant source of trouble for build failures due to host-user-contaminated QA
> > > > errors of sort
> > > >
> > > > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid 3004, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
> > > >
> > > > So far we have tried to mould cp command into not carrying the build
> > > > user permissions into install area but it is never entirely fixed since
> > > > the issue keeps popping up in various scenes
> > > >
> > > > This patch replaces use of cp with install utility and specifies install
> > > > mode for files explcitly
> > > >
> > > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > > ---
> > > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
> > > > 1 file changed, 25 insertions(+), 19 deletions(-)
> > > >
> > > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> > > > index 6384f9cbf1..9b256a5108 100644
> > > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
> > > >
> > > > do_install () {
> > > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > > - fi
> > > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > > - mkdir -p ${D}${localedir}
> > > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > > - fi
> > > > + install -d ${D}${bindir}
> > > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > > > +
> > > > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -printf '%P ') ; do
> > > > + install -d "${D}${localedir}/$d"
> > > > + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1 -type f \
> > > > + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> > > > + done
> > > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > > - mkdir -p ${D}${libdir}
> > > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > > - fi
> > > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > > - fi
> > > > - fi
> > > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > > + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
> > > > + install -d "${D}${libdir}/gconv/$d"
> > > > + find "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
> > > > + -exec install -m 0755 -t "${D}${libdir}/gconv/$d" {} \;
> > > > + done
> > > > + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
> > > > + install -d "${D}${datadir}/i18n/$d"
> > > > + find "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
> > > > + -exec install -m 0644 -t "${D}${datadir}/i18n/$d" {} \;
> > > > + done
> > > > fi
> > > > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type d -printf '%P ') ; do
> > > > + install -d "${D}${datadir}/locale/$d"
> > > > + find "${LOCALETREESRC}/${datadir}/locale/$d" -maxdepth 1 -type f \
> > > > + -exec install -m 0644 -t "${D}${datadir}/locale/$d" {} \;
> > > > + done
> > > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> > > > }
> > > >
> > > > inherit libc-package
> > >
> > > The trouble is this is a workaround. The cp commands should work and
> > > there is some underlying issue in pseudo causing this. We really need
> > > to figure out what that is since this will likely just mean we see the
> > > problem somewhere else :(
> >
> > so far this is the only place where it shows in world builds that
> > meta-oe does which includes many layers.
> >
> > >
> > > I still suspect some kind of inode number reuse problem which these cp
> > > commands trigger...
> >
> > cp and install are not same when it comes to how they operate
> > underneath. install is better when it comes to what we are doing here
> > it also gives better control of permissions and mods,as a recipe writer
> > someone would prefer then to let the tool ( cp ) assume it.
> >
> > So there might be a pseudo issue, I dont disagree but not obvious and
> > I believe using cp is subpar anyway.
> >
> > >
> > > Cheers,
> > >
> > > Richard
> > >
> >
> > --
> >
> > Joshua Watt <JPEWhacker@gmail.com>
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 16:59 ` Joshua Watt
2019-02-07 17:11 ` Martin Jansa
@ 2019-02-07 17:21 ` Richard Purdie
2019-02-07 19:34 ` Joshua Watt
1 sibling, 1 reply; 22+ messages in thread
From: Richard Purdie @ 2019-02-07 17:21 UTC (permalink / raw)
To: Joshua Watt, Martin Jansa, Khem Raj
Cc: Patches and discussions about the oe-core layer
On Thu, 2019-02-07 at 10:59 -0600, Joshua Watt wrote:
> On Thu, Feb 7, 2019 at 10:44 AM Joshua Watt <jpewhacker@gmail.com>
> wrote:
> > Martin,
> >
> > Do you by any chance build in Docker (or some other transient
> > container)? We recently found an issue with building in docker
> > where when the primary container command would exit, causing the
> > container to terminate. When the container terminates, the kernel
> > unceremoniously sends SIGKILL to all its processes. This plays
> > *very* poorly with pseudo because pseudo keeps it's entire database
> > in a in-memory sqlite database and only flushes it out to disk on a
> > periodic interval, or when it get SIGTERM (maybe SIGQUIT?) or exits
> > normally. By default the pseudo daemon will also hang around for
> > 20(?) seconds waiting for a new connection from a pseudo client,
> > meaning that when our docker container was exiting, there were
> > potentially several pseudo dameons sitting around waiting for a
> > client with unflushed databases in memory that suddenly got a
> > SIGKILL and all their data was lost. This was particularly bad if
> > you aborted a build while it was running, or some build failure
> > occurred. Since
> the pseudo database was lost, the only way to correct these errors
> was to clean and rebuild.
> > I think pseudo could handle this better and keep the database on
> > disk instead of in memory, but in the short term you can hack
> > bitbake to pass the -S flag when invoking the pseudo client which
> > will make it tell the dameon to shutdown (and thus flush the
> > database) if it is the last connection.
>
> FWIW, this patch makes it much easier to do that
> http://lists.openembedded.org/pipermail/bitbake-devel/2019-February/019813.html
> Just add
> FAKEROOTCMD += "-S"
> in local.conf
Should we be doing that by default? Is there a downside?
Keeping the database flushed to disk would likely be slow but this
seems reasonable.
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 17:21 ` Richard Purdie
@ 2019-02-07 19:34 ` Joshua Watt
0 siblings, 0 replies; 22+ messages in thread
From: Joshua Watt @ 2019-02-07 19:34 UTC (permalink / raw)
To: Richard Purdie, Martin Jansa, Khem Raj
Cc: Patches and discussions about the oe-core layer
On Thu, 2019-02-07 at 17:21 +0000, Richard Purdie wrote:
> On Thu, 2019-02-07 at 10:59 -0600, Joshua Watt wrote:
> > On Thu, Feb 7, 2019 at 10:44 AM Joshua Watt <jpewhacker@gmail.com>
> > wrote:
> > > Martin,
> > >
> > > Do you by any chance build in Docker (or some other transient
> > > container)? We recently found an issue with building in docker
> > > where when the primary container command would exit, causing the
> > > container to terminate. When the container terminates, the kernel
> > > unceremoniously sends SIGKILL to all its processes. This plays
> > > *very* poorly with pseudo because pseudo keeps it's entire
> > > database
> > > in a in-memory sqlite database and only flushes it out to disk on
> > > a
> > > periodic interval, or when it get SIGTERM (maybe SIGQUIT?) or
> > > exits
> > > normally. By default the pseudo daemon will also hang around for
> > > 20(?) seconds waiting for a new connection from a pseudo client,
> > > meaning that when our docker container was exiting, there were
> > > potentially several pseudo dameons sitting around waiting for a
> > > client with unflushed databases in memory that suddenly got a
> > > SIGKILL and all their data was lost. This was particularly bad if
> > > you aborted a build while it was running, or some build failure
> > > occurred. Since
> > the pseudo database was lost, the only way to correct these errors
> > was to clean and rebuild.
> > > I think pseudo could handle this better and keep the database on
> > > disk instead of in memory, but in the short term you can hack
> > > bitbake to pass the -S flag when invoking the pseudo client which
> > > will make it tell the dameon to shutdown (and thus flush the
> > > database) if it is the last connection.
> >
> > FWIW, this patch makes it much easier to do that
> > http://lists.openembedded.org/pipermail/bitbake-devel/2019-February/019813.html
> > Just add
> > FAKEROOTCMD += "-S"
> > in local.conf
>
> Should we be doing that by default? Is there a downside?
pseudo won't stick around waiting for connections from other clients.
I'm not sure if this would impact performance at all, as I'm not sure
how often a client is able to connect during the "grace period" to
prevent a new server needing to be spun up. I would image this
potentially cuts down on the spin up time for back-to-back fakeroot
tasks in the same recipe?
>
> Keeping the database flushed to disk would likely be slow but this
> seems reasonable.
I think there are 3 options in increasing order of
complexity/risk/safety
1. Add -S to FAKEROOTCMD by default
2. Make pseudo flush the in memory cache every time the last client
disconnects
3. Remove the in-memory cache from pseudo completely and make it write
to disk each time. Even if we removed all disk flushes from the sqlite
database to keep sync()'s down, it would no worse that what happens
today in any case I can think of (specifically, a power loss will cause
lost data), and in many cases it would be better (e.g. SIGKILL and
crashes in pseudo wouldn't cause data loss).
>
> Cheers,
>
> Richard
>
>
--
Joshua Watt <JPEWhacker@gmail.com>
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
@ 2019-02-08 0:56 Khem Raj
2019-02-08 8:44 ` Peter Kjellerstedt
0 siblings, 1 reply; 22+ messages in thread
From: Khem Raj @ 2019-02-08 0:56 UTC (permalink / raw)
To: openembedded-core
This has been a constant source of trouble for build failures due to host-user-contaminated QA
errors of sort
ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid 3004, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
So far we have tried to mould cp command into not carrying the build
user permissions into install area but it is never entirely fixed since
the issue keeps popping up in various scenes
This patch replaces use of cp with install utility and specifies install
mode for files explcitly
Signed-off-by: Khem Raj <raj.khem@gmail.com>
---
meta/recipes-core/glibc/glibc-locale.inc | 41 ++++++++++++------------
1 file changed, 20 insertions(+), 21 deletions(-)
diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
index 6384f9cbf1..9cce61bf0b 100644
--- a/meta/recipes-core/glibc/glibc-locale.inc
+++ b/meta/recipes-core/glibc/glibc-locale.inc
@@ -71,28 +71,27 @@ FILES_localedef = "${bindir}/localedef"
LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
-do_install () {
- mkdir -p ${D}${bindir} ${D}${datadir}
- if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
- fi
- if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
- mkdir -p ${D}${localedir}
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
- fi
- if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
- mkdir -p ${D}${libdir}
- if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
- fi
- if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
- fi
- fi
- if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
+copy_locale_files() {
+ local dir=$1 mode=$2
+
+ [ -e "${LOCALETREESRC}$dir" ] || return 0
+
+ for d in . $(find "${LOCALETREESRC}$dir" -type d -printf '%P '); do
+ install -d ${D}$dir/$d
+ find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
+ -exec install -m $mode -t "${D}$dir/$d" {} \;
+ done
+}
+
+do_install() {
+ copy_locale_files ${bindir} 0755
+ copy_locale_files ${localedir} 0644
+ if [ ${PACKAGE_NO_GCONV} -eq 0 ]; then
+ copy_locale_files ${libdir}/gconv 0755
+ copy_locale_files ${datadir}/i18n 0644
fi
- cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
+ copy_locale_files ${datadir}/locale 0644
+ install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
}
inherit libc-package
--
2.20.1
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-08 0:56 Khem Raj
@ 2019-02-08 8:44 ` Peter Kjellerstedt
2019-02-08 15:21 ` Khem Raj
0 siblings, 1 reply; 22+ messages in thread
From: Peter Kjellerstedt @ 2019-02-08 8:44 UTC (permalink / raw)
To: Khem Raj, openembedded-core@lists.openembedded.org
> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Khem Raj
> Sent: den 8 februari 2019 01:56
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> install utility instead of cp
>
> This has been a constant source of trouble for build failures due to
> host-user-contaminated QA errors of sort
>
> ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid
> 3004, which is the same as the user running bitbake. This may be due to
> host contamination [host-user-contaminated]
>
> So far we have tried to mould cp command into not carrying the build
> user permissions into install area but it is never entirely fixed since
> the issue keeps popping up in various scenes
>
> This patch replaces use of cp with install utility and specifies install
> mode for files explcitly
>
> Signed-off-by: Khem Raj <raj.khem@gmail.com>
> ---
> meta/recipes-core/glibc/glibc-locale.inc | 41 ++++++++++++------------
> 1 file changed, 20 insertions(+), 21 deletions(-)
>
> diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> index 6384f9cbf1..9cce61bf0b 100644
> --- a/meta/recipes-core/glibc/glibc-locale.inc
> +++ b/meta/recipes-core/glibc/glibc-locale.inc
> @@ -71,28 +71,27 @@ FILES_localedef = "${bindir}/localedef"
>
> LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
>
> -do_install () {
> - mkdir -p ${D}${bindir} ${D}${datadir}
> - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> - fi
> - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> - mkdir -p ${D}${localedir}
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> - fi
> - if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> - mkdir -p ${D}${libdir}
> - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> - fi
> - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> - fi
> - fi
> - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> +copy_locale_files() {
> + local dir=$1 mode=$2
> +
> + [ -e "${LOCALETREESRC}$dir" ] || return 0
> +
> + for d in . $(find "${LOCALETREESRC}$dir" -type d -printf '%P '); do
> + install -d ${D}$dir/$d
> + find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
> + -exec install -m $mode -t "${D}$dir/$d" {} \;
I know I am a sucker for details, but please align the broken find line so
that it is more obvious that the second line is a continuation, i.e.:
find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
-exec install -m $mode -t "${D}$dir/$d" {} \;
> + done
> +}
> +
> +do_install() {
> + copy_locale_files ${bindir} 0755
> + copy_locale_files ${localedir} 0644
> + if [ ${PACKAGE_NO_GCONV} -eq 0 ]; then
> + copy_locale_files ${libdir}/gconv 0755
> + copy_locale_files ${datadir}/i18n 0644
> fi
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> + copy_locale_files ${datadir}/locale 0644
> + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> }
>
> inherit libc-package
> --
> 2.20.1
//Peter
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-08 8:44 ` Peter Kjellerstedt
@ 2019-02-08 15:21 ` Khem Raj
2019-02-10 0:34 ` Peter Kjellerstedt
0 siblings, 1 reply; 22+ messages in thread
From: Khem Raj @ 2019-02-08 15:21 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: openembedded-core@lists.openembedded.org
On Fri, Feb 8, 2019 at 12:44 AM Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> > core-bounces@lists.openembedded.org> On Behalf Of Khem Raj
> > Sent: den 8 februari 2019 01:56
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > install utility instead of cp
> >
> > This has been a constant source of trouble for build failures due to
> > host-user-contaminated QA errors of sort
> >
> > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> > es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid
> > 3004, which is the same as the user running bitbake. This may be due to
> > host contamination [host-user-contaminated]
> >
> > So far we have tried to mould cp command into not carrying the build
> > user permissions into install area but it is never entirely fixed since
> > the issue keeps popping up in various scenes
> >
> > This patch replaces use of cp with install utility and specifies install
> > mode for files explcitly
> >
> > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > ---
> > meta/recipes-core/glibc/glibc-locale.inc | 41 ++++++++++++------------
> > 1 file changed, 20 insertions(+), 21 deletions(-)
> >
> > diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> > index 6384f9cbf1..9cce61bf0b 100644
> > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > @@ -71,28 +71,27 @@ FILES_localedef = "${bindir}/localedef"
> >
> > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
> >
> > -do_install () {
> > - mkdir -p ${D}${bindir} ${D}${datadir}
> > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > - fi
> > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > - mkdir -p ${D}${localedir}
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > - fi
> > - if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > - mkdir -p ${D}${libdir}
> > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > - fi
> > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > - fi
> > - fi
> > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > +copy_locale_files() {
> > + local dir=$1 mode=$2
> > +
> > + [ -e "${LOCALETREESRC}$dir" ] || return 0
> > +
> > + for d in . $(find "${LOCALETREESRC}$dir" -type d -printf '%P '); do
> > + install -d ${D}$dir/$d
> > + find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
> > + -exec install -m $mode -t "${D}$dir/$d" {} \;
>
> I know I am a sucker for details, but please align the broken find line so
> that it is more obvious that the second line is a continuation, i.e.:
>
> find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
> -exec install -m $mode -t "${D}$dir/$d" {} \;
>
You are doing good here, I really appreciate the feedback. Do you
think aligning it
under beginning of quote is more readable or aligning where find cmd
begins is better ?
I will make this as a followup patch and send it with next lot of
minor changes that I have in
queue.
> > + done
> > +}
> > +
> > +do_install() {
> > + copy_locale_files ${bindir} 0755
> > + copy_locale_files ${localedir} 0644
> > + if [ ${PACKAGE_NO_GCONV} -eq 0 ]; then
> > + copy_locale_files ${libdir}/gconv 0755
> > + copy_locale_files ${datadir}/i18n 0644
> > fi
> > - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > + copy_locale_files ${datadir}/locale 0644
> > + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> > }
> >
> > inherit libc-package
> > --
> > 2.20.1
>
> //Peter
>
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-08 15:21 ` Khem Raj
@ 2019-02-10 0:34 ` Peter Kjellerstedt
0 siblings, 0 replies; 22+ messages in thread
From: Peter Kjellerstedt @ 2019-02-10 0:34 UTC (permalink / raw)
To: Khem Raj; +Cc: openembedded-core@lists.openembedded.org
> -----Original Message-----
> From: Khem Raj <raj.khem@gmail.com>
> Sent: den 8 februari 2019 16:21
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> install utility instead of cp
>
> On Fri, Feb 8, 2019 at 12:44 AM Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> >
> > > -----Original Message-----
> > > From: openembedded-core-bounces@lists.openembedded.org
> <openembedded-
> > > core-bounces@lists.openembedded.org> On Behalf Of Khem Raj
> > > Sent: den 8 februari 2019 01:56
> > > To: openembedded-core@lists.openembedded.org
> > > Subject: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > > install utility instead of cp
> > >
> > > This has been a constant source of trouble for build failures due
> to
> > > host-user-contaminated QA errors of sort
> > >
> > > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> > > es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by
> uid
> > > 3004, which is the same as the user running bitbake. This may be
> due to
> > > host contamination [host-user-contaminated]
> > >
> > > So far we have tried to mould cp command into not carrying the
> build
> > > user permissions into install area but it is never entirely fixed
> since
> > > the issue keeps popping up in various scenes
> > >
> > > This patch replaces use of cp with install utility and specifies
> install
> > > mode for files explcitly
> > >
> > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > ---
> > > meta/recipes-core/glibc/glibc-locale.inc | 41 ++++++++++++--------
> ----
> > > 1 file changed, 20 insertions(+), 21 deletions(-)
> > >
> > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc
> b/meta/recipes-core/glibc/glibc-locale.inc
> > > index 6384f9cbf1..9cce61bf0b 100644
> > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > @@ -71,28 +71,27 @@ FILES_localedef = "${bindir}/localedef"
> > >
> > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-
> locale"
> > >
> > > -do_install () {
> > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > - fi
> > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > - mkdir -p ${D}${localedir}
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > - fi
> > > - if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > - mkdir -p ${D}${libdir}
> > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > - fi
> > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > - fi
> > > - fi
> > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > +copy_locale_files() {
> > > + local dir=$1 mode=$2
> > > +
> > > + [ -e "${LOCALETREESRC}$dir" ] || return 0
> > > +
> > > + for d in . $(find "${LOCALETREESRC}$dir" -type d -printf '%P
> '); do
> > > + install -d ${D}$dir/$d
> > > + find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
> > > + -exec install -m $mode -t "${D}$dir/$d" {} \;
> >
> > I know I am a sucker for details, but please align the broken find
> line so
> > that it is more obvious that the second line is a continuation, i.e.:
> >
> > find "${LOCALETREESRC}$dir/$d" -maxdepth 1 -type f \
> > -exec install -m $mode -t "${D}$dir/$d" {} \;
> >
>
> You are doing good here, I really appreciate the feedback. Do you
> think aligning it
> under beginning of quote is more readable or aligning where find cmd
> begins is better ?
> I will make this as a followup patch and send it with next lot of
> minor changes that I have in
> queue.
The patch, as it was integrated to master, is fine.
> > > + done
> > > +}
> > > +
> > > +do_install() {
> > > + copy_locale_files ${bindir} 0755
> > > + copy_locale_files ${localedir} 0644
> > > + if [ ${PACKAGE_NO_GCONV} -eq 0 ]; then
> > > + copy_locale_files ${libdir}/gconv 0755
> > > + copy_locale_files ${datadir}/i18n 0644
> > > fi
> > > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > + copy_locale_files ${datadir}/locale 0644
> > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED
> ${WORKDIR}/SUPPORTED
> > > }
> > >
> > > inherit libc-package
> > > --
> > > 2.20.1
> >
> > //Peter
> >
//Peter
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 9:00 ` Richard Purdie
2019-02-07 14:49 ` Khem Raj
@ 2019-02-10 1:29 ` Peter Kjellerstedt
2019-02-10 6:25 ` Peter Kjellerstedt
1 sibling, 1 reply; 22+ messages in thread
From: Peter Kjellerstedt @ 2019-02-10 1:29 UTC (permalink / raw)
To: Richard Purdie, Khem Raj,
openembedded-core@lists.openembedded.org
[-- Attachment #1: Type: text/plain, Size: 6774 bytes --]
> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 7 februari 2019 10:01
> To: Khem Raj <raj.khem@gmail.com>; openembedded-
> core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> install utility instead of cp
>
> On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > This has been a constant source of trouble for build failures due to
> host-user-contaminated QA
> > errors of sort
> >
> > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid
> 3004, which is the same as the user running bitbake. This may be due to
> host contamination [host-user-contaminated]
> >
> > So far we have tried to mould cp command into not carrying the build
> > user permissions into install area but it is never entirely fixed
> since
> > the issue keeps popping up in various scenes
> >
> > This patch replaces use of cp with install utility and specifies
> install
> > mode for files explcitly
> >
> > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > ---
> > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++--------
> --
> > 1 file changed, 25 insertions(+), 19 deletions(-)
> >
> > diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-
> core/glibc/glibc-locale.inc
> > index 6384f9cbf1..9b256a5108 100644
> > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-
> locale"
> >
> > do_install () {
> > - mkdir -p ${D}${bindir} ${D}${datadir}
> > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > - fi
> > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > - mkdir -p ${D}${localedir}
> > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > - fi
> > + install -d ${D}${bindir}
> > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > +
> > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -
> printf '%P ') ; do
> > + install -d "${D}${localedir}/$d"
> > + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1
> -type f \
> > + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> > + done
> > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > - mkdir -p ${D}${libdir}
> > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > - cp -R --no-dereference --
> preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > - fi
> > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > - cp -R --no-dereference --
> preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > - fi
> > - fi
> > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv"
> -type d -printf '%P ') ; do
> > + install -d "${D}${libdir}/gconv/$d"
> > + find "${LOCALETREESRC}/${libdir}/gconv/$d" -
> maxdepth 1 -type f \
> > + -exec install -m 0755 -t
> "${D}${libdir}/gconv/$d" {} \;
> > + done
> > + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n"
> -type d -printf '%P ') ; do
> > + install -d "${D}${datadir}/i18n/$d"
> > + find "${LOCALETREESRC}/${datadir}/i18n/$d" -
> maxdepth 1 -type f \
> > + -exec install -m 0644 -t
> "${D}${datadir}/i18n/$d" {} \;
> > + done
> > fi
> > - cp -R --no-dereference --preserve=mode,links
> ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type
> d -printf '%P ') ; do
> > + install -d "${D}${datadir}/locale/$d"
> > + find "${LOCALETREESRC}/${datadir}/locale/$d" -
> maxdepth 1 -type f \
> > + -exec install -m 0644 -t "${D}${datadir}/locale/$d"
> {} \;
> > + done
> > + install -m 0644 ${LOCALETREESRC}/SUPPORTED
> ${WORKDIR}/SUPPORTED
> > }
> >
> > inherit libc-package
>
> The trouble is this is a workaround. The cp commands should work and
> there is some underlying issue in pseudo causing this. We really need
> to figure out what that is since this will likely just mean we see the
> problem somewhere else :(
>
> I still suspect some kind of inode number reuse problem which these cp
> commands trigger...
>
> Cheers,
>
> Richard
For the record, even if I think the patch was the right thing to do, it
did not help the situation. The first build I did after it was integrated,
I was bit by the following:
Failed to determine the user name of UID 323 for
tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/package/usr/lib/locale
where UID 323 is my UID. If you do not recognize the error message, it is
because it comes from a local patch we have for rpm to protect us from any
incorrect UIDs/GIDs causing incorrect RPMs to be generated (see below). I
checked the build directory and the /usr/lib/locale directory was the only
file or directory with an incorrect owner, which is typical of these
pseudo related problems with incorrect owners.
[rpm will by default silently fall back to use root for any files where it
cannot determine the name of the user or group based on its UID/GID. This
can be due to missing information in the /etc/passwd or /etc/groups files,
typically due to relying on another recipe in DEPENDS to create the user
without also having an RDEPENDS on the package that creates the user, or
because of pseudo messing up. After this bit us once where a device in
/dev ended up in an rpm in the sstate cache with root as group instead of
the intended video group, which in turn caused the video application to
fail as it was no longer allowed to open its device, I created the patch.
It is causing a fair bit of grief as it causes builds to fail randomly,
but at least we don't end up with an incorrect sstate anymore, which is
worse. I am not sure this patch is suitable for integration to OE-Core,
but I have attached it if anyone is interested.]
//Peter
[-- Attachment #2: 0001-rpm-Make-it-an-error-if-a-UID-GID-cannot-be-turned-i.patch --]
[-- Type: application/octet-stream, Size: 3889 bytes --]
From 0f96b74bce4cf071ee5c22c4776f1352a1f43812 Mon Sep 17 00:00:00 2001
From: Peter Kjellerstedt <pkj@axis.com>
Date: Fri, 3 Nov 2017 13:44:40 +0100
Subject: [PATCH] rpm: Make it an error if a UID/GID cannot be turned into a
name
Before, any unknown UID or GID would result in the name of the current
user (typically root) being silently used instead, which leads to
undeterministic behavior depending on whether the required user or
group name is available in /etc/passwd or /etc/group, or not.
This is only appropriate for the native and nativesdk versions.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
...error-if-a-UID-GID-cannot-be-turned-into-.patch | 49 ++++++++++++++++++++++
meta/recipes-devtools/rpm/rpm_4.14.2.1.bb | 2 +
2 files changed, 51 insertions(+)
create mode 100644 meta/recipes-devtools/rpm/files/0001-Make-it-an-error-if-a-UID-GID-cannot-be-turned-into-.patch
diff --git a/meta/recipes-devtools/rpm/files/0001-Make-it-an-error-if-a-UID-GID-cannot-be-turned-into-.patch b/meta/recipes-devtools/rpm/files/0001-Make-it-an-error-if-a-UID-GID-cannot-be-turned-into-.patch
new file mode 100644
index 0000000000..a9467dc298
--- /dev/null
+++ b/meta/recipes-devtools/rpm/files/0001-Make-it-an-error-if-a-UID-GID-cannot-be-turned-into-.patch
@@ -0,0 +1,49 @@
+From 4777dc38c7554787caf9a37575c3e2912783ed23 Mon Sep 17 00:00:00 2001
+From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
+Date: Fri, 3 Nov 2017 13:41:09 +0100
+Subject: [PATCH] Make it an error if a UID/GID cannot be turned into a name
+
+Before, any unknown UID or GID would result in the name of the current
+user (typically root) being silently used instead, which leads to
+undeterministic behavior depending on whether the required user or
+group name is available in /etc/passwd or /etc/group, or not.
+
+Upstream-Status: Inappropriate
+Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
+
+---
+ build/files.c | 18 +++++++++++-------
+ 1 file changed, 11 insertions(+), 7 deletions(-)
+
+diff --git a/build/files.c b/build/files.c
+index 585d73e3f..f92ebd9e9 100644
+--- a/build/files.c
++++ b/build/files.c
+@@ -1373,13 +1373,17 @@ static rpmRC addFile(FileList fl, const char * diskPath,
+ } else {
+ fileGname = rpmugGname(fileGid);
+ }
+-
+- /* Default user/group to builder's user/group */
+- if (fileUname == NULL)
+- fileUname = rpmugUname(getuid());
+- if (fileGname == NULL)
+- fileGname = rpmugGname(getgid());
+-
++
++ /* Fail if the user and/or group name cannot be determined */
++ if (fileUname == NULL) {
++ rpmlog(RPMLOG_ERR, _("Failed to determine the user name of UID %d for %s\n"), fileUid, diskPath);
++ goto exit;
++ }
++ if (fileGname == NULL) {
++ rpmlog(RPMLOG_ERR, _("Failed to determine the group name of GID %d for %s\n"), fileGid, diskPath);
++ goto exit;
++ }
++
+ /* S_XXX macro must be consistent with type in find call at check-files script */
+ if (check_fileList && (S_ISREG(fileMode) || S_ISLNK(fileMode))) {
+ appendStringBuf(check_fileList, diskPath);
+--
+2.12.0
+
diff --git a/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb b/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb
index 063f4269a5..3316c7d60f 100644
--- a/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb
@@ -42,6 +42,8 @@ SRC_URI = "git://github.com/rpm-software-management/rpm;branch=rpm-4.14.x \
file://0001-rpm-rpmio.c-restrict-virtual-memory-usage-if-limit-s.patch \
file://0016-rpmscript.c-change-logging-level-around-scriptlets-t.patch \
"
+SRC_URI_append_class-native = " file://0001-Make-it-an-error-if-a-UID-GID-cannot-be-turned-into-.patch"
+SRC_URI_append_class-nativesdk = " file://0001-Make-it-an-error-if-a-UID-GID-cannot-be-turned-into-.patch"
PE = "1"
SRCREV = "4a9440006398646583f0d9ae1837dad2875013aa"
--
2.12.0
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-10 1:29 ` Peter Kjellerstedt
@ 2019-02-10 6:25 ` Peter Kjellerstedt
2019-02-10 23:32 ` Peter Kjellerstedt
0 siblings, 1 reply; 22+ messages in thread
From: Peter Kjellerstedt @ 2019-02-10 6:25 UTC (permalink / raw)
To: Richard Purdie, Khem Raj,
openembedded-core@lists.openembedded.org
> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> Sent: den 10 februari 2019 02:29
> To: Richard Purdie <richard.purdie@linuxfoundation.org>; Khem Raj
> <raj.khem@gmail.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> install utility instead of cp
>
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> > core-bounces@lists.openembedded.org> On Behalf Of Richard Purdie
> > Sent: den 7 februari 2019 10:01
> > To: Khem Raj <raj.khem@gmail.com>; openembedded-
> > core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > install utility instead of cp
> >
> > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > > This has been a constant source of trouble for build failures due
> to
> > host-user-contaminated QA
> > > errors of sort
> > >
> > > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> > es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid
> > 3004, which is the same as the user running bitbake. This may be due
> to
> > host contamination [host-user-contaminated]
> > >
> > > So far we have tried to mould cp command into not carrying the
> build
> > > user permissions into install area but it is never entirely fixed
> > since
> > > the issue keeps popping up in various scenes
> > >
> > > This patch replaces use of cp with install utility and specifies
> > install
> > > mode for files explcitly
> > >
> > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > ---
> > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++------
> --
> > --
> > > 1 file changed, 25 insertions(+), 19 deletions(-)
> > >
> > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc
> b/meta/recipes-
> > core/glibc/glibc-locale.inc
> > > index 6384f9cbf1..9b256a5108 100644
> > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-
> > locale"
> > >
> > > do_install () {
> > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> > ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > - fi
> > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > - mkdir -p ${D}${localedir}
> > > - cp -R --no-dereference --preserve=mode,links
> > ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > - fi
> > > + install -d ${D}${bindir}
> > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > > +
> > > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d
> -
> > printf '%P ') ; do
> > > + install -d "${D}${localedir}/$d"
> > > + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth
> 1
> > -type f \
> > > + -exec install -m 0644 -t "${D}${localedir}/$d" {}
> \;
> > > + done
> > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > - mkdir -p ${D}${libdir}
> > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > - cp -R --no-dereference --
> > preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > - fi
> > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > - cp -R --no-dereference --
> > preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > - fi
> > > - fi
> > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > - cp -R --no-dereference --preserve=mode,links
> > ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > + for d in . $(find
> "${LOCALETREESRC}/${libdir}/gconv"
> > -type d -printf '%P ') ; do
> > > + install -d "${D}${libdir}/gconv/$d"
> > > + find "${LOCALETREESRC}/${libdir}/gconv/$d"
> -
> > maxdepth 1 -type f \
> > > + -exec install -m 0755 -t
> > "${D}${libdir}/gconv/$d" {} \;
> > > + done
> > > + for d in . $(find
> "${LOCALETREESRC}/${datadir}/i18n"
> > -type d -printf '%P ') ; do
> > > + install -d "${D}${datadir}/i18n/$d"
> > > + find "${LOCALETREESRC}/${datadir}/i18n/$d"
> -
> > maxdepth 1 -type f \
> > > + -exec install -m 0644 -t
> > "${D}${datadir}/i18n/$d" {} \;
> > > + done
> > > fi
> > > - cp -R --no-dereference --preserve=mode,links
> > ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -
> type
> > d -printf '%P ') ; do
> > > + install -d "${D}${datadir}/locale/$d"
> > > + find "${LOCALETREESRC}/${datadir}/locale/$d" -
> > maxdepth 1 -type f \
> > > + -exec install -m 0644 -t
> "${D}${datadir}/locale/$d"
> > {} \;
> > > + done
> > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED
> > ${WORKDIR}/SUPPORTED
> > > }
> > >
> > > inherit libc-package
> >
> > The trouble is this is a workaround. The cp commands should work and
> > there is some underlying issue in pseudo causing this. We really need
> > to figure out what that is since this will likely just mean we see
> the
> > problem somewhere else :(
> >
> > I still suspect some kind of inode number reuse problem which these
> cp
> > commands trigger...
> >
> > Cheers,
> >
> > Richard
>
> For the record, even if I think the patch was the right thing to do,
> it did not help the situation. The first build I did after it was
> integrated, I was bit by the following:
>
> Failed to determine the user name of UID 323 for
> tmp/work/core2-64-poky-linux/glibc-locale/2.29-r0/package/usr/lib/locale
>
> where UID 323 is my UID. If you do not recognize the error message,
> it is because it comes from a local patch we have for rpm to protect
> us from any incorrect UIDs/GIDs causing incorrect RPMs to be generated
> (see below). I checked the build directory and the /usr/lib/locale
> directory was the only file or directory with an incorrect owner,
> which is typical of these pseudo related problems with incorrect
> owners.
>
> [rpm will by default silently fall back to use root for any files
> where it cannot determine the name of the user or group based on
> its UID/GID. This can be due to missing information in the
> /etc/passwd or /etc/groups files, typically due to relying on
> another recipe in DEPENDS to create the user without also having
> an RDEPENDS on the package that creates the user, or because of
> pseudo messing up. After this bit us once where a device in /dev
> ended up in an rpm in the sstate cache with root as group instead
> of the intended video group, which in turn caused the video
> application to fail as it was no longer allowed to open its device,
> I created the patch. It is causing a fair bit of grief as it causes
> builds to fail randomly, but at least we don't end up with an
> incorrect sstate anymore, which is worse. I am not sure this patch
> is suitable for integration to OE-Core, but I have attached it if
> anyone is interested.]
>
> //Peter
Ok, this just got a lot more weird. At the time I wrote the mail
above, I had only tried to rebuild once. Now I have done it a number
of more times, and the result is not random any more. Every build
fails the same way. I've done `bitbake glibc-locale -c cleansstate`
followed by `bitbake glibc-locale` and the result is consistent.
What I have noted now, is that it is _not_ do_install that causes
the problem. The /usr/lib/locale directory created in ${D} has
the correct owner. It is when the data is copied to ${PKGD} that
it somehow ends up with the wrong owner. I cannot tell from a quick
glance what differs /usr/lib/locale from, e.g., /usr/lib/gconv,
where the latter does get the correct owner.
Unfortunately, I do not have more time to look at this right now,
but I thought I should share this information with you. If anyone
has any ideas of what to look at to determine why
${PKGD}/usr/lib/locale ends up with the wrong owner, I'm all ears.
It would also be interesting to hear if the result is the same for
anyone else with the patch I provided in my previous mail applied.
//Peter
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-10 6:25 ` Peter Kjellerstedt
@ 2019-02-10 23:32 ` Peter Kjellerstedt
2019-02-11 2:09 ` Khem Raj
0 siblings, 1 reply; 22+ messages in thread
From: Peter Kjellerstedt @ 2019-02-10 23:32 UTC (permalink / raw)
To: Richard Purdie, Khem Raj,
openembedded-core@lists.openembedded.org
> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> Sent: den 10 februari 2019 07:25
> To: Richard Purdie <richard.purdie@linuxfoundation.org>; Khem Raj
> <raj.khem@gmail.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> install utility instead of cp
>
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> > core-bounces@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> > Sent: den 10 februari 2019 02:29
> > To: Richard Purdie <richard.purdie@linuxfoundation.org>; Khem Raj
> > <raj.khem@gmail.com>; openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > install utility instead of cp
> >
> > > -----Original Message-----
> > > From: openembedded-core-bounces@lists.openembedded.org
> <openembedded-
> > > core-bounces@lists.openembedded.org> On Behalf Of Richard Purdie
> > > Sent: den 7 februari 2019 10:01
> > > To: Khem Raj <raj.khem@gmail.com>; openembedded-
> > > core@lists.openembedded.org
> > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install
> using
> > > install utility instead of cp
> > >
> > > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > > > This has been a constant source of trouble for build failures due
> > to
> > > host-user-contaminated QA
> > > > errors of sort
> > > >
> > > > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> > > es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by
> uid
> > > 3004, which is the same as the user running bitbake. This may be
> due
> > to
> > > host contamination [host-user-contaminated]
> > > >
> > > > So far we have tried to mould cp command into not carrying the
> > build
> > > > user permissions into install area but it is never entirely fixed
> > > since
> > > > the issue keeps popping up in various scenes
> > > >
> > > > This patch replaces use of cp with install utility and specifies
> > > install
> > > > mode for files explcitly
> > > >
> > > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > > ---
> > > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----
> --
> > --
> > > --
> > > > 1 file changed, 25 insertions(+), 19 deletions(-)
> > > >
> > > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc
> > b/meta/recipes-
> > > core/glibc/glibc-locale.inc
> > > > index 6384f9cbf1..9b256a5108 100644
> > > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-
> > > locale"
> > > >
> > > > do_install () {
> > > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > > - cp -R --no-dereference --preserve=mode,links
> > > ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > > - fi
> > > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > > - mkdir -p ${D}${localedir}
> > > > - cp -R --no-dereference --preserve=mode,links
> > > ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > > - fi
> > > > + install -d ${D}${bindir}
> > > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > > > +
> > > > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type
> d
> > -
> > > printf '%P ') ; do
> > > > + install -d "${D}${localedir}/$d"
> > > > + find "${LOCALETREESRC}/${localedir}/$d" -
> maxdepth
> > 1
> > > -type f \
> > > > + -exec install -m 0644 -t "${D}${localedir}/$d"
> {}
> > \;
> > > > + done
> > > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > > - mkdir -p ${D}${libdir}
> > > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > > - cp -R --no-dereference --
> > > preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > > - fi
> > > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > > - cp -R --no-dereference --
> > > preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > > - fi
> > > > - fi
> > > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > > - cp -R --no-dereference --preserve=mode,links
> > > ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > > + for d in . $(find
> > "${LOCALETREESRC}/${libdir}/gconv"
> > > -type d -printf '%P ') ; do
> > > > + install -d "${D}${libdir}/gconv/$d"
> > > > + find
> "${LOCALETREESRC}/${libdir}/gconv/$d"
> > -
> > > maxdepth 1 -type f \
> > > > + -exec install -m 0755 -t
> > > "${D}${libdir}/gconv/$d" {} \;
> > > > + done
> > > > + for d in . $(find
> > "${LOCALETREESRC}/${datadir}/i18n"
> > > -type d -printf '%P ') ; do
> > > > + install -d "${D}${datadir}/i18n/$d"
> > > > + find
> "${LOCALETREESRC}/${datadir}/i18n/$d"
> > -
> > > maxdepth 1 -type f \
> > > > + -exec install -m 0644 -t
> > > "${D}${datadir}/i18n/$d" {} \;
> > > > + done
> > > > fi
> > > > - cp -R --no-dereference --preserve=mode,links
> > > ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -
> > type
> > > d -printf '%P ') ; do
> > > > + install -d "${D}${datadir}/locale/$d"
> > > > + find "${LOCALETREESRC}/${datadir}/locale/$d" -
> > > maxdepth 1 -type f \
> > > > + -exec install -m 0644 -t
> > "${D}${datadir}/locale/$d"
> > > {} \;
> > > > + done
> > > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED
> > > ${WORKDIR}/SUPPORTED
> > > > }
> > > >
> > > > inherit libc-package
> > >
> > > The trouble is this is a workaround. The cp commands should work
> and
> > > there is some underlying issue in pseudo causing this. We really
> need
> > > to figure out what that is since this will likely just mean we see
> > the
> > > problem somewhere else :(
> > >
> > > I still suspect some kind of inode number reuse problem which these
> > cp
> > > commands trigger...
> > >
> > > Cheers,
> > >
> > > Richard
> >
> > For the record, even if I think the patch was the right thing to do,
> > it did not help the situation. The first build I did after it was
> > integrated, I was bit by the following:
> >
> > Failed to determine the user name of UID 323 for
> > tmp/work/core2-64-poky-linux/glibc-locale/2.29-
> r0/package/usr/lib/locale
> >
> > where UID 323 is my UID. If you do not recognize the error message,
> > it is because it comes from a local patch we have for rpm to protect
> > us from any incorrect UIDs/GIDs causing incorrect RPMs to be
> generated
> > (see below). I checked the build directory and the /usr/lib/locale
> > directory was the only file or directory with an incorrect owner,
> > which is typical of these pseudo related problems with incorrect
> > owners.
> >
> > [rpm will by default silently fall back to use root for any files
> > where it cannot determine the name of the user or group based on
> > its UID/GID. This can be due to missing information in the
> > /etc/passwd or /etc/groups files, typically due to relying on
> > another recipe in DEPENDS to create the user without also having
> > an RDEPENDS on the package that creates the user, or because of
> > pseudo messing up. After this bit us once where a device in /dev
> > ended up in an rpm in the sstate cache with root as group instead
> > of the intended video group, which in turn caused the video
> > application to fail as it was no longer allowed to open its device,
> > I created the patch. It is causing a fair bit of grief as it causes
> > builds to fail randomly, but at least we don't end up with an
> > incorrect sstate anymore, which is worse. I am not sure this patch
> > is suitable for integration to OE-Core, but I have attached it if
> > anyone is interested.]
> >
> > //Peter
>
> Ok, this just got a lot more weird. At the time I wrote the mail
> above, I had only tried to rebuild once. Now I have done it a number
> of more times, and the result is not random any more. Every build
> fails the same way. I've done `bitbake glibc-locale -c cleansstate`
> followed by `bitbake glibc-locale` and the result is consistent.
> What I have noted now, is that it is _not_ do_install that causes
> the problem. The /usr/lib/locale directory created in ${D} has
> the correct owner. It is when the data is copied to ${PKGD} that
> it somehow ends up with the wrong owner. I cannot tell from a quick
> glance what differs /usr/lib/locale from, e.g., /usr/lib/gconv,
> where the latter does get the correct owner.
>
> Unfortunately, I do not have more time to look at this right now,
> but I thought I should share this information with you. If anyone
> has any ideas of what to look at to determine why
> ${PKGD}/usr/lib/locale ends up with the wrong owner, I'm all ears.
> It would also be interesting to hear if the result is the same for
> anyone else with the patch I provided in my previous mail applied.
>
> //Peter
Further investigation indicates that the problem I'm seeing is not
related to pseudo, but an actual problem in the code. I have not dug
too far into it, but I'm seeing the problem in the directories used
in do_prep_locale_tree() and do_collect_bins_from_locale_tree() in
libc-package.bbclass where some of them seems to have been created
without being run under pseudo. The following patch works around the
problem, though it would probably be better to correct the problem
at the source by running whatever part of the code under pseudo that
is currently missing it:
diff --git a/meta/classes/libc-package.bbclass b/meta/classes/libc-package.bbclass
index 0b4c666a74..174465a891 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -72,6 +72,7 @@ do_prep_locale_tree() {
tar -cf - -C ${STAGING_DIR_NATIVE}/${prefix_native}/${base_libdir} -p libgcc_s.* | tar -xf - -C $treedir/${base_libdir}
fi
install -m 0755 ${LOCALETREESRC}${bindir}/localedef $treedir/${base_bindir}
+ chown -R root:root $treedir
}
do_collect_bins_from_locale_tree() {
//Peter
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-10 23:32 ` Peter Kjellerstedt
@ 2019-02-11 2:09 ` Khem Raj
2019-02-13 17:42 ` Khem Raj
0 siblings, 1 reply; 22+ messages in thread
From: Khem Raj @ 2019-02-11 2:09 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: openembedded-core@lists.openembedded.org
On Sun, Feb 10, 2019 at 3:32 PM Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> > -----Original Message-----
> > From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> > core-bounces@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> > Sent: den 10 februari 2019 07:25
> > To: Richard Purdie <richard.purdie@linuxfoundation.org>; Khem Raj
> > <raj.khem@gmail.com>; openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > install utility instead of cp
> >
> > > -----Original Message-----
> > > From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> > > core-bounces@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> > > Sent: den 10 februari 2019 02:29
> > > To: Richard Purdie <richard.purdie@linuxfoundation.org>; Khem Raj
> > > <raj.khem@gmail.com>; openembedded-core@lists.openembedded.org
> > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > > install utility instead of cp
> > >
> > > > -----Original Message-----
> > > > From: openembedded-core-bounces@lists.openembedded.org
> > <openembedded-
> > > > core-bounces@lists.openembedded.org> On Behalf Of Richard Purdie
> > > > Sent: den 7 februari 2019 10:01
> > > > To: Khem Raj <raj.khem@gmail.com>; openembedded-
> > > > core@lists.openembedded.org
> > > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install
> > using
> > > > install utility instead of cp
> > > >
> > > > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > > > > This has been a constant source of trouble for build failures due
> > > to
> > > > host-user-contaminated QA
> > > > > errors of sort
> > > > >
> > > > > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> > > > es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by
> > uid
> > > > 3004, which is the same as the user running bitbake. This may be
> > due
> > > to
> > > > host contamination [host-user-contaminated]
> > > > >
> > > > > So far we have tried to mould cp command into not carrying the
> > > build
> > > > > user permissions into install area but it is never entirely fixed
> > > > since
> > > > > the issue keeps popping up in various scenes
> > > > >
> > > > > This patch replaces use of cp with install utility and specifies
> > > > install
> > > > > mode for files explcitly
> > > > >
> > > > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > > > ---
> > > > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----
> > --
> > > --
> > > > --
> > > > > 1 file changed, 25 insertions(+), 19 deletions(-)
> > > > >
> > > > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc
> > > b/meta/recipes-
> > > > core/glibc/glibc-locale.inc
> > > > > index 6384f9cbf1..9b256a5108 100644
> > > > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > > > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-
> > > > locale"
> > > > >
> > > > > do_install () {
> > > > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > > > - cp -R --no-dereference --preserve=mode,links
> > > > ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > > > - fi
> > > > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > > > - mkdir -p ${D}${localedir}
> > > > > - cp -R --no-dereference --preserve=mode,links
> > > > ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > > > - fi
> > > > > + install -d ${D}${bindir}
> > > > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > > > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > > > > +
> > > > > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type
> > d
> > > -
> > > > printf '%P ') ; do
> > > > > + install -d "${D}${localedir}/$d"
> > > > > + find "${LOCALETREESRC}/${localedir}/$d" -
> > maxdepth
> > > 1
> > > > -type f \
> > > > > + -exec install -m 0644 -t "${D}${localedir}/$d"
> > {}
> > > \;
> > > > > + done
> > > > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > > > - mkdir -p ${D}${libdir}
> > > > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > > > - cp -R --no-dereference --
> > > > preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > > > - fi
> > > > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > > > - cp -R --no-dereference --
> > > > preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > > > - fi
> > > > > - fi
> > > > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > > > - cp -R --no-dereference --preserve=mode,links
> > > > ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > > > + for d in . $(find
> > > "${LOCALETREESRC}/${libdir}/gconv"
> > > > -type d -printf '%P ') ; do
> > > > > + install -d "${D}${libdir}/gconv/$d"
> > > > > + find
> > "${LOCALETREESRC}/${libdir}/gconv/$d"
> > > -
> > > > maxdepth 1 -type f \
> > > > > + -exec install -m 0755 -t
> > > > "${D}${libdir}/gconv/$d" {} \;
> > > > > + done
> > > > > + for d in . $(find
> > > "${LOCALETREESRC}/${datadir}/i18n"
> > > > -type d -printf '%P ') ; do
> > > > > + install -d "${D}${datadir}/i18n/$d"
> > > > > + find
> > "${LOCALETREESRC}/${datadir}/i18n/$d"
> > > -
> > > > maxdepth 1 -type f \
> > > > > + -exec install -m 0644 -t
> > > > "${D}${datadir}/i18n/$d" {} \;
> > > > > + done
> > > > > fi
> > > > > - cp -R --no-dereference --preserve=mode,links
> > > > ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -
> > > type
> > > > d -printf '%P ') ; do
> > > > > + install -d "${D}${datadir}/locale/$d"
> > > > > + find "${LOCALETREESRC}/${datadir}/locale/$d" -
> > > > maxdepth 1 -type f \
> > > > > + -exec install -m 0644 -t
> > > "${D}${datadir}/locale/$d"
> > > > {} \;
> > > > > + done
> > > > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED
> > > > ${WORKDIR}/SUPPORTED
> > > > > }
> > > > >
> > > > > inherit libc-package
> > > >
> > > > The trouble is this is a workaround. The cp commands should work
> > and
> > > > there is some underlying issue in pseudo causing this. We really
> > need
> > > > to figure out what that is since this will likely just mean we see
> > > the
> > > > problem somewhere else :(
> > > >
> > > > I still suspect some kind of inode number reuse problem which these
> > > cp
> > > > commands trigger...
> > > >
> > > > Cheers,
> > > >
> > > > Richard
> > >
> > > For the record, even if I think the patch was the right thing to do,
> > > it did not help the situation. The first build I did after it was
> > > integrated, I was bit by the following:
> > >
> > > Failed to determine the user name of UID 323 for
> > > tmp/work/core2-64-poky-linux/glibc-locale/2.29-
> > r0/package/usr/lib/locale
> > >
> > > where UID 323 is my UID. If you do not recognize the error message,
> > > it is because it comes from a local patch we have for rpm to protect
> > > us from any incorrect UIDs/GIDs causing incorrect RPMs to be
> > generated
> > > (see below). I checked the build directory and the /usr/lib/locale
> > > directory was the only file or directory with an incorrect owner,
> > > which is typical of these pseudo related problems with incorrect
> > > owners.
> > >
> > > [rpm will by default silently fall back to use root for any files
> > > where it cannot determine the name of the user or group based on
> > > its UID/GID. This can be due to missing information in the
> > > /etc/passwd or /etc/groups files, typically due to relying on
> > > another recipe in DEPENDS to create the user without also having
> > > an RDEPENDS on the package that creates the user, or because of
> > > pseudo messing up. After this bit us once where a device in /dev
> > > ended up in an rpm in the sstate cache with root as group instead
> > > of the intended video group, which in turn caused the video
> > > application to fail as it was no longer allowed to open its device,
> > > I created the patch. It is causing a fair bit of grief as it causes
> > > builds to fail randomly, but at least we don't end up with an
> > > incorrect sstate anymore, which is worse. I am not sure this patch
> > > is suitable for integration to OE-Core, but I have attached it if
> > > anyone is interested.]
> > >
> > > //Peter
> >
> > Ok, this just got a lot more weird. At the time I wrote the mail
> > above, I had only tried to rebuild once. Now I have done it a number
> > of more times, and the result is not random any more. Every build
> > fails the same way. I've done `bitbake glibc-locale -c cleansstate`
> > followed by `bitbake glibc-locale` and the result is consistent.
> > What I have noted now, is that it is _not_ do_install that causes
> > the problem. The /usr/lib/locale directory created in ${D} has
> > the correct owner. It is when the data is copied to ${PKGD} that
> > it somehow ends up with the wrong owner. I cannot tell from a quick
> > glance what differs /usr/lib/locale from, e.g., /usr/lib/gconv,
> > where the latter does get the correct owner.
> >
> > Unfortunately, I do not have more time to look at this right now,
> > but I thought I should share this information with you. If anyone
> > has any ideas of what to look at to determine why
> > ${PKGD}/usr/lib/locale ends up with the wrong owner, I'm all ears.
> > It would also be interesting to hear if the result is the same for
> > anyone else with the patch I provided in my previous mail applied.
> >
> > //Peter
>
> Further investigation indicates that the problem I'm seeing is not
> related to pseudo, but an actual problem in the code. I have not dug
> too far into it, but I'm seeing the problem in the directories used
> in do_prep_locale_tree() and do_collect_bins_from_locale_tree() in
> libc-package.bbclass where some of them seems to have been created
> without being run under pseudo. The following patch works around the
> problem, though it would probably be better to correct the problem
> at the source by running whatever part of the code under pseudo that
> is currently missing it:
>
> diff --git a/meta/classes/libc-package.bbclass b/meta/classes/libc-package.bbclass
> index 0b4c666a74..174465a891 100644
> --- a/meta/classes/libc-package.bbclass
> +++ b/meta/classes/libc-package.bbclass
> @@ -72,6 +72,7 @@ do_prep_locale_tree() {
> tar -cf - -C ${STAGING_DIR_NATIVE}/${prefix_native}/${base_libdir} -p libgcc_s.* | tar -xf - -C $treedir/${base_libdir}
> fi
> install -m 0755 ${LOCALETREESRC}${bindir}/localedef $treedir/${base_bindir}
> + chown -R root:root $treedir
> }
>
> do_collect_bins_from_locale_tree() {
>
Can you try something on the lines
diff --git a/meta/classes/libc-package.bbclass
b/meta/classes/libc-package.bbclass
index 0b4c666a74..34025635ce 100644
--- a/meta/classes/libc-package.bbclass
+++ b/meta/classes/libc-package.bbclass
@@ -61,7 +61,7 @@ LOCALETREESRC ?= "${PKGD}"
do_prep_locale_tree() {
treedir=${WORKDIR}/locale-tree
rm -rf $treedir
- mkdir -p $treedir/${base_bindir} $treedir/${base_libdir}
$treedir/${datadir} $treedir/${localedir}
+ install -d $treedir/${base_bindir} $treedir/${base_libdir}
$treedir/${datadir} $treedir/${localedir}
tar -cf - -C ${LOCALETREESRC}${datadir} -p i18n | tar -xf - -C
$treedir/${datadir}
# unzip to avoid parsing errors
for i in $treedir/${datadir}/i18n/charmaps/*gz; do
> //Peter
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-11 2:09 ` Khem Raj
@ 2019-02-13 17:42 ` Khem Raj
0 siblings, 0 replies; 22+ messages in thread
From: Khem Raj @ 2019-02-13 17:42 UTC (permalink / raw)
To: Peter Kjellerstedt; +Cc: openembedded-core@lists.openembedded.org
Peter did you have a chance to try
https://patchwork.openembedded.org/patch/158661/
On Sun, Feb 10, 2019 at 6:09 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Sun, Feb 10, 2019 at 3:32 PM Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> >
> > > -----Original Message-----
> > > From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> > > core-bounces@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> > > Sent: den 10 februari 2019 07:25
> > > To: Richard Purdie <richard.purdie@linuxfoundation.org>; Khem Raj
> > > <raj.khem@gmail.com>; openembedded-core@lists.openembedded.org
> > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > > install utility instead of cp
> > >
> > > > -----Original Message-----
> > > > From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> > > > core-bounces@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> > > > Sent: den 10 februari 2019 02:29
> > > > To: Richard Purdie <richard.purdie@linuxfoundation.org>; Khem Raj
> > > > <raj.khem@gmail.com>; openembedded-core@lists.openembedded.org
> > > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > > > install utility instead of cp
> > > >
> > > > > -----Original Message-----
> > > > > From: openembedded-core-bounces@lists.openembedded.org
> > > <openembedded-
> > > > > core-bounces@lists.openembedded.org> On Behalf Of Richard Purdie
> > > > > Sent: den 7 februari 2019 10:01
> > > > > To: Khem Raj <raj.khem@gmail.com>; openembedded-
> > > > > core@lists.openembedded.org
> > > > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install
> > > using
> > > > > install utility instead of cp
> > > > >
> > > > > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > > > > > This has been a constant source of trouble for build failures due
> > > > to
> > > > > host-user-contaminated QA
> > > > > > errors of sort
> > > > > >
> > > > > > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> > > > > es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by
> > > uid
> > > > > 3004, which is the same as the user running bitbake. This may be
> > > due
> > > > to
> > > > > host contamination [host-user-contaminated]
> > > > > >
> > > > > > So far we have tried to mould cp command into not carrying the
> > > > build
> > > > > > user permissions into install area but it is never entirely fixed
> > > > > since
> > > > > > the issue keeps popping up in various scenes
> > > > > >
> > > > > > This patch replaces use of cp with install utility and specifies
> > > > > install
> > > > > > mode for files explcitly
> > > > > >
> > > > > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > > > > ---
> > > > > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----
> > > --
> > > > --
> > > > > --
> > > > > > 1 file changed, 25 insertions(+), 19 deletions(-)
> > > > > >
> > > > > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc
> > > > b/meta/recipes-
> > > > > core/glibc/glibc-locale.inc
> > > > > > index 6384f9cbf1..9b256a5108 100644
> > > > > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > > > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > > > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > > > > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-
> > > > > locale"
> > > > > >
> > > > > > do_install () {
> > > > > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > > > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > > > > - cp -R --no-dereference --preserve=mode,links
> > > > > ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > > > > - fi
> > > > > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > > > > - mkdir -p ${D}${localedir}
> > > > > > - cp -R --no-dereference --preserve=mode,links
> > > > > ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > > > > - fi
> > > > > > + install -d ${D}${bindir}
> > > > > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > > > > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > > > > > +
> > > > > > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type
> > > d
> > > > -
> > > > > printf '%P ') ; do
> > > > > > + install -d "${D}${localedir}/$d"
> > > > > > + find "${LOCALETREESRC}/${localedir}/$d" -
> > > maxdepth
> > > > 1
> > > > > -type f \
> > > > > > + -exec install -m 0644 -t "${D}${localedir}/$d"
> > > {}
> > > > \;
> > > > > > + done
> > > > > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > > > > - mkdir -p ${D}${libdir}
> > > > > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > > > > - cp -R --no-dereference --
> > > > > preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > > > > - fi
> > > > > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > > > > - cp -R --no-dereference --
> > > > > preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > > > > - fi
> > > > > > - fi
> > > > > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > > > > - cp -R --no-dereference --preserve=mode,links
> > > > > ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > > > > + for d in . $(find
> > > > "${LOCALETREESRC}/${libdir}/gconv"
> > > > > -type d -printf '%P ') ; do
> > > > > > + install -d "${D}${libdir}/gconv/$d"
> > > > > > + find
> > > "${LOCALETREESRC}/${libdir}/gconv/$d"
> > > > -
> > > > > maxdepth 1 -type f \
> > > > > > + -exec install -m 0755 -t
> > > > > "${D}${libdir}/gconv/$d" {} \;
> > > > > > + done
> > > > > > + for d in . $(find
> > > > "${LOCALETREESRC}/${datadir}/i18n"
> > > > > -type d -printf '%P ') ; do
> > > > > > + install -d "${D}${datadir}/i18n/$d"
> > > > > > + find
> > > "${LOCALETREESRC}/${datadir}/i18n/$d"
> > > > -
> > > > > maxdepth 1 -type f \
> > > > > > + -exec install -m 0644 -t
> > > > > "${D}${datadir}/i18n/$d" {} \;
> > > > > > + done
> > > > > > fi
> > > > > > - cp -R --no-dereference --preserve=mode,links
> > > > > ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > > > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -
> > > > type
> > > > > d -printf '%P ') ; do
> > > > > > + install -d "${D}${datadir}/locale/$d"
> > > > > > + find "${LOCALETREESRC}/${datadir}/locale/$d" -
> > > > > maxdepth 1 -type f \
> > > > > > + -exec install -m 0644 -t
> > > > "${D}${datadir}/locale/$d"
> > > > > {} \;
> > > > > > + done
> > > > > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED
> > > > > ${WORKDIR}/SUPPORTED
> > > > > > }
> > > > > >
> > > > > > inherit libc-package
> > > > >
> > > > > The trouble is this is a workaround. The cp commands should work
> > > and
> > > > > there is some underlying issue in pseudo causing this. We really
> > > need
> > > > > to figure out what that is since this will likely just mean we see
> > > > the
> > > > > problem somewhere else :(
> > > > >
> > > > > I still suspect some kind of inode number reuse problem which these
> > > > cp
> > > > > commands trigger...
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Richard
> > > >
> > > > For the record, even if I think the patch was the right thing to do,
> > > > it did not help the situation. The first build I did after it was
> > > > integrated, I was bit by the following:
> > > >
> > > > Failed to determine the user name of UID 323 for
> > > > tmp/work/core2-64-poky-linux/glibc-locale/2.29-
> > > r0/package/usr/lib/locale
> > > >
> > > > where UID 323 is my UID. If you do not recognize the error message,
> > > > it is because it comes from a local patch we have for rpm to protect
> > > > us from any incorrect UIDs/GIDs causing incorrect RPMs to be
> > > generated
> > > > (see below). I checked the build directory and the /usr/lib/locale
> > > > directory was the only file or directory with an incorrect owner,
> > > > which is typical of these pseudo related problems with incorrect
> > > > owners.
> > > >
> > > > [rpm will by default silently fall back to use root for any files
> > > > where it cannot determine the name of the user or group based on
> > > > its UID/GID. This can be due to missing information in the
> > > > /etc/passwd or /etc/groups files, typically due to relying on
> > > > another recipe in DEPENDS to create the user without also having
> > > > an RDEPENDS on the package that creates the user, or because of
> > > > pseudo messing up. After this bit us once where a device in /dev
> > > > ended up in an rpm in the sstate cache with root as group instead
> > > > of the intended video group, which in turn caused the video
> > > > application to fail as it was no longer allowed to open its device,
> > > > I created the patch. It is causing a fair bit of grief as it causes
> > > > builds to fail randomly, but at least we don't end up with an
> > > > incorrect sstate anymore, which is worse. I am not sure this patch
> > > > is suitable for integration to OE-Core, but I have attached it if
> > > > anyone is interested.]
> > > >
> > > > //Peter
> > >
> > > Ok, this just got a lot more weird. At the time I wrote the mail
> > > above, I had only tried to rebuild once. Now I have done it a number
> > > of more times, and the result is not random any more. Every build
> > > fails the same way. I've done `bitbake glibc-locale -c cleansstate`
> > > followed by `bitbake glibc-locale` and the result is consistent.
> > > What I have noted now, is that it is _not_ do_install that causes
> > > the problem. The /usr/lib/locale directory created in ${D} has
> > > the correct owner. It is when the data is copied to ${PKGD} that
> > > it somehow ends up with the wrong owner. I cannot tell from a quick
> > > glance what differs /usr/lib/locale from, e.g., /usr/lib/gconv,
> > > where the latter does get the correct owner.
> > >
> > > Unfortunately, I do not have more time to look at this right now,
> > > but I thought I should share this information with you. If anyone
> > > has any ideas of what to look at to determine why
> > > ${PKGD}/usr/lib/locale ends up with the wrong owner, I'm all ears.
> > > It would also be interesting to hear if the result is the same for
> > > anyone else with the patch I provided in my previous mail applied.
> > >
> > > //Peter
> >
> > Further investigation indicates that the problem I'm seeing is not
> > related to pseudo, but an actual problem in the code. I have not dug
> > too far into it, but I'm seeing the problem in the directories used
> > in do_prep_locale_tree() and do_collect_bins_from_locale_tree() in
> > libc-package.bbclass where some of them seems to have been created
> > without being run under pseudo. The following patch works around the
> > problem, though it would probably be better to correct the problem
> > at the source by running whatever part of the code under pseudo that
> > is currently missing it:
> >
> > diff --git a/meta/classes/libc-package.bbclass b/meta/classes/libc-package.bbclass
> > index 0b4c666a74..174465a891 100644
> > --- a/meta/classes/libc-package.bbclass
> > +++ b/meta/classes/libc-package.bbclass
> > @@ -72,6 +72,7 @@ do_prep_locale_tree() {
> > tar -cf - -C ${STAGING_DIR_NATIVE}/${prefix_native}/${base_libdir} -p libgcc_s.* | tar -xf - -C $treedir/${base_libdir}
> > fi
> > install -m 0755 ${LOCALETREESRC}${bindir}/localedef $treedir/${base_bindir}
> > + chown -R root:root $treedir
> > }
> >
> > do_collect_bins_from_locale_tree() {
> >
>
>
> Can you try something on the lines
>
> diff --git a/meta/classes/libc-package.bbclass
> b/meta/classes/libc-package.bbclass
> index 0b4c666a74..34025635ce 100644
> --- a/meta/classes/libc-package.bbclass
> +++ b/meta/classes/libc-package.bbclass
> @@ -61,7 +61,7 @@ LOCALETREESRC ?= "${PKGD}"
> do_prep_locale_tree() {
> treedir=${WORKDIR}/locale-tree
> rm -rf $treedir
> - mkdir -p $treedir/${base_bindir} $treedir/${base_libdir}
> $treedir/${datadir} $treedir/${localedir}
> + install -d $treedir/${base_bindir} $treedir/${base_libdir}
> $treedir/${datadir} $treedir/${localedir}
> tar -cf - -C ${LOCALETREESRC}${datadir} -p i18n | tar -xf - -C
> $treedir/${datadir}
> # unzip to avoid parsing errors
> for i in $treedir/${datadir}/i18n/charmaps/*gz; do
>
>
>
> > //Peter
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-07 0:35 [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp Khem Raj
2019-02-07 9:00 ` Richard Purdie
2019-02-07 11:44 ` Peter Kjellerstedt
@ 2019-02-28 1:53 ` ChenQi
2019-02-28 15:21 ` Richard Purdie
2 siblings, 1 reply; 22+ messages in thread
From: ChenQi @ 2019-02-28 1:53 UTC (permalink / raw)
To: Khem Raj, openembedded-core
Just to provide some feedback.
Even with this patch, this annoying QA issue is still there.
WARNING: glibc-locale-2.29-r0 do_package_qa: QA Issue: glibc-locale:
/glibc-binary-localedata-wo-sn/usr/lib/locale/wo_SN/LC_CTYPE is owned by
uid 1001, which is the same as the user running bitbake. This may be due
to host contamination [host-user-contaminated]
Best Regards,
Chen Qi
On 02/07/2019 08:35 AM, Khem Raj wrote:
> This has been a constant source of trouble for build failures due to host-user-contaminated QA
> errors of sort
>
> ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-es+valencia/usr/lib/locale/ca_ES@valencia/LC_MONETARY is owned by uid 3004, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
>
> So far we have tried to mould cp command into not carrying the build
> user permissions into install area but it is never entirely fixed since
> the issue keeps popping up in various scenes
>
> This patch replaces use of cp with install utility and specifies install
> mode for files explcitly
>
> Signed-off-by: Khem Raj <raj.khem@gmail.com>
> ---
> meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----------
> 1 file changed, 25 insertions(+), 19 deletions(-)
>
> diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
> index 6384f9cbf1..9b256a5108 100644
> --- a/meta/recipes-core/glibc/glibc-locale.inc
> +++ b/meta/recipes-core/glibc/glibc-locale.inc
> @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
>
> do_install () {
> - mkdir -p ${D}${bindir} ${D}${datadir}
> - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> - fi
> - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> - mkdir -p ${D}${localedir}
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> - fi
> + install -d ${D}${bindir}
> + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> + -exec install -m 0755 -t "${D}${bindir}" {} \;
> +
> + for d in . $(find "${LOCALETREESRC}/${localedir}" -type d -printf '%P ') ; do
> + install -d "${D}${localedir}/$d"
> + find "${LOCALETREESRC}/${localedir}/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${localedir}/$d" {} \;
> + done
> if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> - mkdir -p ${D}${libdir}
> - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> - fi
> - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> - fi
> - fi
> - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> + for d in . $(find "${LOCALETREESRC}/${libdir}/gconv" -type d -printf '%P ') ; do
> + install -d "${D}${libdir}/gconv/$d"
> + find "${LOCALETREESRC}/${libdir}/gconv/$d" -maxdepth 1 -type f \
> + -exec install -m 0755 -t "${D}${libdir}/gconv/$d" {} \;
> + done
> + for d in . $(find "${LOCALETREESRC}/${datadir}/i18n" -type d -printf '%P ') ; do
> + install -d "${D}${datadir}/i18n/$d"
> + find "${LOCALETREESRC}/${datadir}/i18n/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${datadir}/i18n/$d" {} \;
> + done
> fi
> - cp -R --no-dereference --preserve=mode,links ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -type d -printf '%P ') ; do
> + install -d "${D}${datadir}/locale/$d"
> + find "${LOCALETREESRC}/${datadir}/locale/$d" -maxdepth 1 -type f \
> + -exec install -m 0644 -t "${D}${datadir}/locale/$d" {} \;
> + done
> + install -m 0644 ${LOCALETREESRC}/SUPPORTED ${WORKDIR}/SUPPORTED
> }
>
> inherit libc-package
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
2019-02-28 1:53 ` ChenQi
@ 2019-02-28 15:21 ` Richard Purdie
0 siblings, 0 replies; 22+ messages in thread
From: Richard Purdie @ 2019-02-28 15:21 UTC (permalink / raw)
To: ChenQi, Khem Raj, openembedded-core
On Thu, 2019-02-28 at 09:53 +0800, ChenQi wrote:
> Just to provide some feedback.
> Even with this patch, this annoying QA issue is still there.
> WARNING: glibc-locale-2.29-r0 do_package_qa: QA Issue: glibc-locale:
> /glibc-binary-localedata-wo-sn/usr/lib/locale/wo_SN/LC_CTYPE is owned
> by
> uid 1001, which is the same as the user running bitbake. This may be
> due
> to host contamination [host-user-contaminated]
Just seen on our autobuilder too:
WARNING: glibc-locale-2.29-r0 do_package_qa: QA Issue: glibc-locale: /glibc-binary-localedata-gez-et+abegede/usr/lib/locale/gez_ET@abegede/LC_MESSAGES/SYS_LC_MESSAGES is owned by uid 6000, which is the same as the user running bitbake. This may be due to host contamination [host-user-contaminated]
(https://autobuilder.yoctoproject.org/typhoon/#/builders/70/builds/346)
Cheers,
Richard
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2019-02-28 15:21 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-02-07 0:35 [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp Khem Raj
2019-02-07 9:00 ` Richard Purdie
2019-02-07 14:49 ` Khem Raj
2019-02-07 16:17 ` Martin Jansa
2019-02-07 16:44 ` Joshua Watt
2019-02-07 16:59 ` Joshua Watt
2019-02-07 17:11 ` Martin Jansa
2019-02-07 17:21 ` Richard Purdie
2019-02-07 19:34 ` Joshua Watt
2019-02-10 1:29 ` Peter Kjellerstedt
2019-02-10 6:25 ` Peter Kjellerstedt
2019-02-10 23:32 ` Peter Kjellerstedt
2019-02-11 2:09 ` Khem Raj
2019-02-13 17:42 ` Khem Raj
2019-02-07 11:44 ` Peter Kjellerstedt
2019-02-07 14:53 ` Khem Raj
2019-02-28 1:53 ` ChenQi
2019-02-28 15:21 ` Richard Purdie
-- strict thread matches above, loose matches on Subject: below --
2019-02-08 0:56 Khem Raj
2019-02-08 8:44 ` Peter Kjellerstedt
2019-02-08 15:21 ` Khem Raj
2019-02-10 0:34 ` Peter Kjellerstedt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox