[PATCH] lib/oe/package: fix LOCALE_PATHS scan to create locale packages

[PATCH] lib/oe/package: fix LOCALE_PATHS scan to create locale packages

[Jonathan GUILLOT]

split_locales() must only check subdirectories in paths added to
LOCALE_PATHS to avoid creating weird packages based on filenames also
present in paths.

Without such a filter, cups recipe adding ${datadir}/cups/templates to
LOCALE_PATHS creates the following incorrect packages:
- cups-locale-add-class.tmpl
- cups-locale-add-printer.tmpl
- cups-locale-admin.tmpl

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
---
 meta/conf/documentation.conf |  2 +-
 meta/lib/oe/package.py       | 14 +++++++++-----
 2 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 90d8e82932..b0591881ba 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -271,7 +271,7 @@ LICENSE_PATH[doc] = "Path to additional licenses used during the build."
 LINUX_KERNEL_TYPE[doc] = "Defines the kernel type to be used in assembling the configuration."
 LINUX_VERSION[doc] = "The Linux version from kernel.org on which the Linux kernel image being built using the OpenEmbedded build system is based. You define this variable in the kernel recipe."
 LINUX_VERSION_EXTENSION[doc] = "A string extension compiled into the version string of the Linux kernel built with the OpenEmbedded build system. You define this variable in the kernel recipe."
-LOCALE_PATHS[doc] = "Whitespace separated list of paths that are scanned to construct locale packages. The list already contains ${datadir}/locale by default."
+LOCALE_PATHS[doc] = "Whitespace separated list of paths that are scanned to construct locale packages. The list already contains ${datadir}/locale by default. Note that all subdirectories in these paths are assumed to be locales."
 LOCALE_UTF8_IS_DEFAULT[doc] = "If set, locale names are renamed such that those lacking an explicit encoding (e.g. en_US) will always be UTF-8, and non-UTF-8 encodings are renamed to, e.g., en_US.ISO-8859-1. Otherwise, the encoding is specified by glibc's SUPPORTED file. Not supported for precompiled locales."
 LOG_DIR[doc] = "Specifies the directory to which the OpenEmbedded build system writes overall log files. The default directory is ${TMPDIR}/log"
 
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index d1738d3b61..587810bdaf 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -652,11 +652,15 @@ def split_locales(d):
     locales = set()
     for localepath in (d.getVar('LOCALE_PATHS') or "").split():
         localedir = dvar + localepath
-        if cpath.isdir(localedir):
-            locales.update(os.listdir(localedir))
-            localepaths.append(localepath)
-        else:
-            bb.debug(1, "No locale files in %s" % localepath)
+        if not cpath.isdir(localedir):
+            bb.debug(1, 'No locale files in %s' % localepath)
+            continue
+
+        localepaths.append(localepath)
+        with os.scandir(localedir) as it:
+            for entry in it:
+                if entry.is_dir():
+                    locales.add(entry.name)
 
     if len(locales) == 0:
         bb.debug(1, "No locale files in this package")

base-commit: f5e6183b9557477bef74024a587de0bfcc2b7c0d
-- 
2.43.0

Re: [OE-core] [PATCH] lib/oe/package: fix LOCALE_PATHS scan to create locale packages

[Richard Purdie]

On Mon, 2024-02-19 at 16:19 +0000, Jonathan GUILLOT wrote:
> split_locales() must only check subdirectories in paths added to
> LOCALE_PATHS to avoid creating weird packages based on filenames also
> present in paths.
> 
> Without such a filter, cups recipe adding ${datadir}/cups/templates
> to
> LOCALE_PATHS creates the following incorrect packages:
> - cups-locale-add-class.tmpl
> - cups-locale-add-printer.tmpl
> - cups-locale-admin.tmpl
> 
> Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
> ---
>  meta/conf/documentation.conf |  2 +-
>  meta/lib/oe/package.py       | 14 +++++++++-----
>  2 files changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/conf/documentation.conf
> b/meta/conf/documentation.conf
> index 90d8e82932..b0591881ba 100644
> --- a/meta/conf/documentation.conf
> +++ b/meta/conf/documentation.conf
> @@ -271,7 +271,7 @@ LICENSE_PATH[doc] = "Path to additional licenses
> used during the build."
>  LINUX_KERNEL_TYPE[doc] = "Defines the kernel type to be used in
> assembling the configuration."
>  LINUX_VERSION[doc] = "The Linux version from kernel.org on which the
> Linux kernel image being built using the OpenEmbedded build system is
> based. You define this variable in the kernel recipe."
>  LINUX_VERSION_EXTENSION[doc] = "A string extension compiled into the
> version string of the Linux kernel built with the OpenEmbedded build
> system. You define this variable in the kernel recipe."
> -LOCALE_PATHS[doc] = "Whitespace separated list of paths that are
> scanned to construct locale packages. The list already contains
> ${datadir}/locale by default."
> +LOCALE_PATHS[doc] = "Whitespace separated list of paths that are
> scanned to construct locale packages. The list already contains
> ${datadir}/locale by default. Note that all subdirectories in these
> paths are assumed to be locales."
>  LOCALE_UTF8_IS_DEFAULT[doc] = "If set, locale names are renamed such
> that those lacking an explicit encoding (e.g. en_US) will always be
> UTF-8, and non-UTF-8 encodings are renamed to, e.g., en_US.ISO-8859-
> 1. Otherwise, the encoding is specified by glibc's SUPPORTED file.
> Not supported for precompiled locales."
>  LOG_DIR[doc] = "Specifies the directory to which the OpenEmbedded
> build system writes overall log files. The default directory is
> ${TMPDIR}/log"
>  
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index d1738d3b61..587810bdaf 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -652,11 +652,15 @@ def split_locales(d):
>      locales = set()
>      for localepath in (d.getVar('LOCALE_PATHS') or "").split():
>          localedir = dvar + localepath
> -        if cpath.isdir(localedir):
> -            locales.update(os.listdir(localedir))
> -            localepaths.append(localepath)
> -        else:
> -            bb.debug(1, "No locale files in %s" % localepath)
> +        if not cpath.isdir(localedir):
> +            bb.debug(1, 'No locale files in %s' % localepath)
> +            continue
> +
> +        localepaths.append(localepath)
> +        with os.scandir(localedir) as it:
> +            for entry in it:
> +                if entry.is_dir():
> +                    locales.add(entry.name)
>  
>      if len(locales) == 0:
>          bb.debug(1, "No locale files in this package")
> 
> base-commit: f5e6183b9557477bef74024a587de0bfcc2b7c0d

Unfortunately this breaks glibc-locale:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6379/steps/14/logs/stdio

ERROR: glibc-locale-2.39+git-r0 do_package: QA Issue: glibc-locale: Files/directories were installed but not shipped in any package:
  /usr/share/locale/locale.alias
Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.

Cheers,

Richard

Re: [OE-core] [PATCH] lib/oe/package: fix LOCALE_PATHS scan to create locale packages

[Jonathan GUILLOT]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

Without any of my modifications, it seems that glibc-locale recipe creates
a glibc-locale-locale.alias package containing only the file
/usr/share/locale/locale.alias. It looks like the same issue revealed with
the CUPS recipe. Do you know in which package locale.alias file is expected
to be located?

Regards,
Jonathan

Le mar. 20 févr. 2024 à 08:42, Richard Purdie <
richard.purdie@linuxfoundation.org> a écrit :

> ERROR: glibc-locale-2.39+git-r0 do_package: QA Issue: glibc-locale:
> Files/directories were installed but not shipped in any package:
>   /usr/share/locale/locale.alias
> Please set FILES such that these items are packaged. Alternatively if they
> are unneeded, avoid installing them or delete them within do_install.
>

[-- Attachment #2: Type: text/html, Size: 1098 bytes --]

Re: [OE-core] [PATCH] lib/oe/package: fix LOCALE_PATHS scan to create locale packages

[Jonathan GUILLOT]

[-- Attachment #1.1: Type: text/plain, Size: 1068 bytes --]

I have attached a patch (as an example) recreating the same
glibc-locale-locale.alias package but it is really weird to me. Is there a
glibc locale expert to help?

Regards,
Jonathan

Le mar. 20 févr. 2024 à 10:45, Jonathan GUILLOT <jonathan@joggee.fr> a
écrit :

> Without any of my modifications, it seems that glibc-locale recipe creates
> a glibc-locale-locale.alias package containing only the file
> /usr/share/locale/locale.alias. It looks like the same issue revealed with
> the CUPS recipe. Do you know in which package locale.alias file is expected
> to be located?
>
> Regards,
> Jonathan
>
> Le mar. 20 févr. 2024 à 08:42, Richard Purdie <
> richard.purdie@linuxfoundation.org> a écrit :
>
>> ERROR: glibc-locale-2.39+git-r0 do_package: QA Issue: glibc-locale:
>> Files/directories were installed but not shipped in any package:
>>   /usr/share/locale/locale.alias
>> Please set FILES such that these items are packaged. Alternatively if
>> they are unneeded, avoid installing them or delete them within do_install.
>>
>

[-- Attachment #1.2: Type: text/html, Size: 1714 bytes --]

[-- Attachment #2: glibc-locale.patch --]
[-- Type: application/octet-stream, Size: 942 bytes --]

diff --git a/meta/recipes-core/glibc/glibc-locale.inc b/meta/recipes-core/glibc/glibc-locale.inc
index 289f58d4df..5e7abaa1c6 100644
--- a/meta/recipes-core/glibc/glibc-locale.inc
+++ b/meta/recipes-core/glibc/glibc-locale.inc
@@ -28,7 +28,7 @@ LOCALE_GENERATION_WITH_CROSS-LOCALEDEF = "1"
 
 PROVIDES = "virtual/libc-locale"
 
-PACKAGES = "localedef ${PN}-dbg"
+PACKAGES = "localedef ${PN}-dbg ${PN}-locale-locale.alias"
 
 PACKAGES_DYNAMIC = "^locale-base-.* \
                     ^glibc-gconv-.* ^glibc-charmap-.* ^glibc-localedata-.* ^glibc-binary-localedata-.* \
@@ -61,6 +61,7 @@ DESCRIPTION:localedef = "glibc: compile locale definition files"
 # Explicitly add ${MLPREFIX} for FILES:glibc-gconv.
 FILES:${MLPREFIX}glibc-gconv = "${libdir}/gconv/*"
 FILES:localedef = "${bindir}/localedef"
+FILES:${PN}-locale-locale.alias = "${datadir}/locale/locale.alias"
 
 LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-locale"
 

Re: [OE-core] [PATCH] lib/oe/package: fix LOCALE_PATHS scan to create locale packages

[Richard Purdie]

On Thu, 2024-02-22 at 11:56 +0100, Jonathan GUILLOT wrote:
> I have attached a patch (as an example) recreating the same glibc-
> locale-locale.alias package but it is really weird to me. Is there a
> glibc locale expert to help?

I had a look at the situation and I think the best thing to do is
package this into a "glibc-locale-alias" package. I think the file
itself is obsolete and not really used so moving to a more logically
named package just in case anyone needs it is probably the bets way
forward.

Cheers,

Richard