[PATCH v4 0/3] lib/oe/package: allow to scan extra directories for locale files

[PATCH v4 0/3] lib/oe/package: allow to scan extra directories for locale files

[Jonathan GUILLOT]

Hi,

Following advice from Ross Burton, here is a new version of the patches to allow scanning extra directories for locale files.

Just to remember the full story, playing with cups I observe the main package includes localized HTML templates in ${datadir}/cups/templates. Locale packages were only looking for files in ${datadir}/locale. This serie of patches allows to add extra directories to be scanned.

Following Ross comment, now the directories to be scanned for locales are defined in the LOCALE_PATHS variable containing ${datadir}/locale by default. Recipes may then add extra paths to this list or even overrides it completely e.g. cups now adds ${datadir}/cups/templates.

Regards,
Jonathan GUILLOT

Jonathan GUILLOT (3):
  lib/oe/package: replace in place PN-locale-* packages in PACKAGES
  lib/oe/package: add LOCALE_PATHS to add define all locations for
    locales
  cups: use LOCALE_PATHS to split localized HTML templates

 meta/classes-global/package.bbclass |  1 +
 meta/conf/documentation.conf        |  1 +
 meta/lib/oe/package.py              | 36 ++++++++++++++++++-----------
 meta/recipes-extended/cups/cups.inc |  2 ++
 4 files changed, 26 insertions(+), 14 deletions(-)


base-commit: b0fdf0688109ba6b87840a0837bebee3d9a27089
-- 
2.43.0

[PATCH v4 1/3] lib/oe/package: replace in place PN-locale-* packages in PACKAGES

[Jonathan GUILLOT]

split_locales() removes PN-locale from PACKAGES and adds PN-locale-* to the end.
As the PN-locale package typically appears before PN base package, it may result
in paths not installed in PN-locale-* packages if already catched by PN. Now
insert PN-locale-* exactly where PN-locale was existing in list to avoid such
an issue.

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
---
 meta/lib/oe/package.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 702d8403be..cb6893f3c2 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -647,8 +647,11 @@ def split_locales(d):
     dvar = d.getVar('PKGD')
     pn = d.getVar('LOCALEBASEPN')
 
-    if pn + '-locale' in packages:
-        packages.remove(pn + '-locale')
+    try:
+        locale_index = packages.index(pn + '-locale')
+        packages.pop(locale_index)
+    except ValueError:
+        locale_index = len(packages)
 
     localedir = os.path.join(dvar + datadir, 'locale')
 
@@ -665,7 +668,8 @@ def split_locales(d):
     for l in sorted(locales):
         ln = legitimize_package_name(l)
         pkg = pn + '-locale-' + ln
-        packages.append(pkg)
+        packages.insert(locale_index, pkg)
+        locale_index += 1
         d.setVar('FILES:' + pkg, os.path.join(datadir, 'locale', l))
         d.setVar('RRECOMMENDS:' + pkg, '%svirtual-locale-%s' % (mlprefix, ln))
         d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln))
-- 
2.43.0

[PATCH v4 2/3] lib/oe/package: add LOCALE_PATHS to add define all locations for locales

[Jonathan GUILLOT]

Some packages may contain localized files not located in default path
${datadir}/locale. Add the new variable LOCALE_PATHS to allow a recipe
to define extra paths or even fully override the scanned directories.
LOCALE_PATHS is set at ${datadir}/locale by default to keep the exact
same behavior for the recipes which did not need modification.

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
---
 meta/classes-global/package.bbclass |  1 +
 meta/conf/documentation.conf        |  1 +
 meta/lib/oe/package.py              | 24 ++++++++++++++----------
 3 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass
index f56bca3542..aa1eb5e901 100644
--- a/meta/classes-global/package.bbclass
+++ b/meta/classes-global/package.bbclass
@@ -335,6 +335,7 @@ python package_convert_pr_autoinc() {
 }
 
 LOCALEBASEPN ??= "${PN}"
+LOCALE_PATHS ?= "${datadir}/locale"
 
 python package_do_split_locales() {
     oe.package.split_locales(d)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 486c62b6e8..90d8e82932 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -271,6 +271,7 @@ LICENSE_PATH[doc] = "Path to additional licenses used during the build."
 LINUX_KERNEL_TYPE[doc] = "Defines the kernel type to be used in assembling the configuration."
 LINUX_VERSION[doc] = "The Linux version from kernel.org on which the Linux kernel image being built using the OpenEmbedded build system is based. You define this variable in the kernel recipe."
 LINUX_VERSION_EXTENSION[doc] = "A string extension compiled into the version string of the Linux kernel built with the OpenEmbedded build system. You define this variable in the kernel recipe."
+LOCALE_PATHS[doc] = "Whitespace separated list of paths that are scanned to construct locale packages. The list already contains ${datadir}/locale by default."
 LOCALE_UTF8_IS_DEFAULT[doc] = "If set, locale names are renamed such that those lacking an explicit encoding (e.g. en_US) will always be UTF-8, and non-UTF-8 encodings are renamed to, e.g., en_US.ISO-8859-1. Otherwise, the encoding is specified by glibc's SUPPORTED file. Not supported for precompiled locales."
 LOG_DIR[doc] = "Specifies the directory to which the OpenEmbedded build system writes overall log files. The default directory is ${TMPDIR}/log"
 
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index cb6893f3c2..d1738d3b61 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -639,11 +639,6 @@ def split_locales(d):
 
     packages = (d.getVar('PACKAGES') or "").split()
 
-    datadir = d.getVar('datadir')
-    if not datadir:
-        bb.note("datadir not defined")
-        return
-
     dvar = d.getVar('PKGD')
     pn = d.getVar('LOCALEBASEPN')
 
@@ -653,14 +648,20 @@ def split_locales(d):
     except ValueError:
         locale_index = len(packages)
 
-    localedir = os.path.join(dvar + datadir, 'locale')
+    localepaths = []
+    locales = set()
+    for localepath in (d.getVar('LOCALE_PATHS') or "").split():
+        localedir = dvar + localepath
+        if cpath.isdir(localedir):
+            locales.update(os.listdir(localedir))
+            localepaths.append(localepath)
+        else:
+            bb.debug(1, "No locale files in %s" % localepath)
 
-    if not cpath.isdir(localedir):
+    if len(locales) == 0:
         bb.debug(1, "No locale files in this package")
         return
 
-    locales = os.listdir(localedir)
-
     summary = d.getVar('SUMMARY') or pn
     description = d.getVar('DESCRIPTION') or ""
     locale_section = d.getVar('LOCALE_SECTION')
@@ -670,7 +671,10 @@ def split_locales(d):
         pkg = pn + '-locale-' + ln
         packages.insert(locale_index, pkg)
         locale_index += 1
-        d.setVar('FILES:' + pkg, os.path.join(datadir, 'locale', l))
+        files = []
+        for localepath in localepaths:
+            files.append(os.path.join(localepath, l))
+        d.setVar('FILES:' + pkg, " ".join(files))
         d.setVar('RRECOMMENDS:' + pkg, '%svirtual-locale-%s' % (mlprefix, ln))
         d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln))
         d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
-- 
2.43.0

[PATCH v4 3/3] cups: use LOCALE_PATHS to split localized HTML templates

[Jonathan GUILLOT]

Localized HTML templates in /usr/share/cups/templates are now part of
locale packages.

Signed-off-by: Jonathan GUILLOT <jonathan@joggee.fr>
---
 meta/recipes-extended/cups/cups.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-extended/cups/cups.inc b/meta/recipes-extended/cups/cups.inc
index 31f686cdfd..b70ba3ae58 100644
--- a/meta/recipes-extended/cups/cups.inc
+++ b/meta/recipes-extended/cups/cups.inc
@@ -110,6 +110,8 @@ CONFFILES:${PN} += "${sysconfdir}/cups/cupsd.conf"
 
 MULTILIB_SCRIPTS = "${PN}-dev:${bindir}/cups-config"
 
+LOCALE_PATHS += "${datadir}/cups/templates"
+
 SYSROOT_PREPROCESS_FUNCS += "cups_sysroot_preprocess"
 cups_sysroot_preprocess () {
 	sed -i ${SYSROOT_DESTDIR}${bindir_crossscripts}/cups-config -e 's:cups_datadir=.*:cups_datadir=${datadir}/cups:' -e 's:cups_serverbin=.*:cups_serverbin=${libexecdir}/cups:'
-- 
2.43.0

Re: [OE-core] [PATCH v4 3/3] cups: use LOCALE_PATHS to split localized HTML templates

[Ross Burton]

On 6 Dec 2023, at 16:06, Jonathan GUILLOT via lists.openembedded.org <jonathan=joggee.fr@lists.openembedded.org> wrote:
> +LOCALE_PATHS += "${datadir}/cups/templates”

This doesn’t quite do what you expect. Listing the packages generated by cups I see:

cups-locale-da:
        /usr/share/cups/templates/da/add-class.tmpl
        /usr/share/cups/templates/da/add-printer.tmpl
        /usr/share/cups/templates/da/admin.tmpl

Which is good.   But I also see:

cups-locale-add-class.tmpl:
        /usr/share/cups/templates/add-class.tmpl
cups-locale-add-printer.tmpl:
        /usr/share/cups/templates/add-printer.tmpl
cups-locale-admin.tmpl:
        /usr/share/cups/templates/admin.tmpl

That doesn’t look right!

Ross

Re: [OE-core] [PATCH v4 3/3] cups: use LOCALE_PATHS to split localized HTML templates

[Jonathan GUILLOT]

[-- Attachment #1: Type: text/plain, Size: 1532 bytes --]

Thanks Ross,

I was too focus on English and French packages and missed this point.
Listing all items in CUPS added locale path is not OK. I should at least
only get directories (not files) in this path to create the locale list.
However, it remains an issue if some directories in the path would not be
dedicated to locales. This is not the point in CUPS but could be in an
other recipe. Do you think it is possible to determine if the directory
name is a valid language code? Is it reliable to consider locale
directories will always be : 2-letter ISO 639 code followed by optional
hyphen and ISO 3166 country code?

Regards,
Jonathan GUILLOT

Le jeu. 15 févr. 2024 à 17:27, Ross Burton <Ross.Burton@arm.com> a écrit :

> On 6 Dec 2023, at 16:06, Jonathan GUILLOT via lists.openembedded.org
> <jonathan=joggee.fr@lists.openembedded.org> wrote:
> > +LOCALE_PATHS += "${datadir}/cups/templates”
>
> This doesn’t quite do what you expect. Listing the packages generated by
> cups I see:
>
> cups-locale-da:
>         /usr/share/cups/templates/da/add-class.tmpl
>         /usr/share/cups/templates/da/add-printer.tmpl
>         /usr/share/cups/templates/da/admin.tmpl
>
> Which is good.   But I also see:
>
> cups-locale-add-class.tmpl:
>         /usr/share/cups/templates/add-class.tmpl
> cups-locale-add-printer.tmpl:
>         /usr/share/cups/templates/add-printer.tmpl
> cups-locale-admin.tmpl:
>         /usr/share/cups/templates/admin.tmpl
>
> That doesn’t look right!
>
> Ross

[-- Attachment #2: Type: text/html, Size: 2100 bytes --]

Re: [OE-core] [PATCH v4 3/3] cups: use LOCALE_PATHS to split localized HTML templates

[Richard Purdie]

On Mon, 2024-02-19 at 11:00 +0100, Jonathan GUILLOT wrote:
> Thanks Ross,
> 
> I was too focus on English and French packages and missed this point.
> Listing all items in CUPS added locale path is not OK. I should at
> least only get directories (not files) in this path to create the
> locale list. However, it remains an issue if some directories in the
> path would not be dedicated to locales. This is not the point in CUPS
> but could be in an other recipe. Do you think it is possible to
> determine if the directory name is a valid language code? Is it
> reliable to consider locale directories will always be : 2-letter ISO
> 639 code followed by optional hyphen and ISO 3166 country code?

Fixing cups is the priority right now. I was hoping for a fix rather
than having to revert things.

Fixing the core code to only handle directories would also be good. We
should probably just note that all subdirectories are assumed to be
locales in a comment where LOCALE_PATHS is given it's default.

Cheers,

Richard