[PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Guénaël Muller]

Modify locale installation method to be standard accross all debian-based distributions.
Pre-existing method is available only on Ubuntu, locale-gen tool has no parameter in Debian.

Signed-off-by: Guénaël Muller <guenael.muller@smile.fr>
Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
Tested-by: Antonin Godard <antonin.godard@bootlin.com>
---
 documentation/poky.yaml.in                       | 3 +--
 documentation/ref-manual/system-requirements.rst | 8 ++++++++
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
index c770318f2..66f758fe8 100644
--- a/documentation/poky.yaml.in
+++ b/documentation/poky.yaml.in
@@ -13,8 +13,7 @@ YOCTO_RELEASE_DL_URL : "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
 UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip texinfo gcc \
      build-essential chrpath socat cpio python3 python3-pip python3-pexpect \
      xz-utils debianutils iputils-ping python3-git python3-jinja2 \
-     python3-subunit zstd liblz4-tool file locales libacl1
-     \n\   $ sudo locale-gen en_US.UTF-8"
+     python3-subunit zstd liblz4-tool file locales libacl1"
 FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip python3 unzip perl patch \
      diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
      ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat \
diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst
index 0fc92550a..b33518c7f 100644
--- a/documentation/ref-manual/system-requirements.rst
+++ b/documentation/ref-manual/system-requirements.rst
@@ -152,6 +152,14 @@ with a supported Ubuntu or Debian Linux distribution::
 
    $ sudo apt install &UBUNTU_HOST_PACKAGES_ESSENTIAL;
 
+You also need to ensure you have the locale ``en_US.UTF-8`` enabled::
+
+   $ locale --all-locales | grep en_US.utf8
+
+If this is not the case, you can reconfigure the ``locales`` package to add it::
+
+   $ sudo dpkg-reconfigure locales
+
 .. note::
 
    -  If your build system has the ``oss4-dev`` package installed, you
-- 
2.39.5

Re: [PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Antonin Godard]

Hi Guénaël,

On Mon Nov 18, 2024 at 11:45 AM CET, Guénaël Muller wrote:
> Modify locale installation method to be standard accross all debian-based distributions.
> Pre-existing method is available only on Ubuntu, locale-gen tool has no parameter in Debian.
>
> Signed-off-by: Guénaël Muller <guenael.muller@smile.fr>
> Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
> Tested-by: Antonin Godard <antonin.godard@bootlin.com>
> ---
>  documentation/poky.yaml.in                       | 3 +--
>  documentation/ref-manual/system-requirements.rst | 8 ++++++++
>  2 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
> index c770318f2..66f758fe8 100644
> --- a/documentation/poky.yaml.in
> +++ b/documentation/poky.yaml.in
> @@ -13,8 +13,7 @@ YOCTO_RELEASE_DL_URL : "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
>  UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip texinfo gcc \
>       build-essential chrpath socat cpio python3 python3-pip python3-pexpect \
>       xz-utils debianutils iputils-ping python3-git python3-jinja2 \
> -     python3-subunit zstd liblz4-tool file locales libacl1
> -     \n\   $ sudo locale-gen en_US.UTF-8"
> +     python3-subunit zstd liblz4-tool file locales libacl1"
>  FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip python3 unzip perl patch \
>       diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
>       ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat \
> diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst
> index 0fc92550a..b33518c7f 100644
> --- a/documentation/ref-manual/system-requirements.rst
> +++ b/documentation/ref-manual/system-requirements.rst
> @@ -152,6 +152,14 @@ with a supported Ubuntu or Debian Linux distribution::
>  
>     $ sudo apt install &UBUNTU_HOST_PACKAGES_ESSENTIAL;
>  
> +You also need to ensure you have the locale ``en_US.UTF-8`` enabled::
> +
> +   $ locale --all-locales | grep en_US.utf8
> +
> +If this is not the case, you can reconfigure the ``locales`` package to add it::
> +
> +   $ sudo dpkg-reconfigure locales
> +
>  .. note::
>  
>     -  If your build system has the ``oss4-dev`` package installed, you

Looks good, thanks!

Reviewed-by: Antonin Godard <antonin.godard@bootlin.com>

Cheers,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Quentin Schulz]

Hi Anotnin, Guénaël,

On 11/18/24 12:00 PM, Antonin Godard via lists.yoctoproject.org wrote:
> Hi Guénaël,
> 
> On Mon Nov 18, 2024 at 11:45 AM CET, Guénaël Muller wrote:
>> Modify locale installation method to be standard accross all debian-based distributions.
>> Pre-existing method is available only on Ubuntu, locale-gen tool has no parameter in Debian.
>>
>> Signed-off-by: Guénaël Muller <guenael.muller@smile.fr>
>> Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
>> Tested-by: Antonin Godard <antonin.godard@bootlin.com>
>> ---
>>   documentation/poky.yaml.in                       | 3 +--
>>   documentation/ref-manual/system-requirements.rst | 8 ++++++++
>>   2 files changed, 9 insertions(+), 2 deletions(-)
>>
>> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
>> index c770318f2..66f758fe8 100644
>> --- a/documentation/poky.yaml.in
>> +++ b/documentation/poky.yaml.in
>> @@ -13,8 +13,7 @@ YOCTO_RELEASE_DL_URL : "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
>>   UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip texinfo gcc \
>>        build-essential chrpath socat cpio python3 python3-pip python3-pexpect \
>>        xz-utils debianutils iputils-ping python3-git python3-jinja2 \
>> -     python3-subunit zstd liblz4-tool file locales libacl1
>> -     \n\   $ sudo locale-gen en_US.UTF-8"
>> +     python3-subunit zstd liblz4-tool file locales libacl1"
>>   FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip python3 unzip perl patch \
>>        diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
>>        ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat \
>> diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst
>> index 0fc92550a..b33518c7f 100644
>> --- a/documentation/ref-manual/system-requirements.rst
>> +++ b/documentation/ref-manual/system-requirements.rst
>> @@ -152,6 +152,14 @@ with a supported Ubuntu or Debian Linux distribution::
>>   
>>      $ sudo apt install &UBUNTU_HOST_PACKAGES_ESSENTIAL;
>>   
>> +You also need to ensure you have the locale ``en_US.UTF-8`` enabled::
>> +

I believe "the ``en_US.UTF-8`` locale" is more proper here.

On a side note, what's the actual cost of reconfiguring the locales 
regardless of the presence of en_US.UTF-8?

My point being we could probably have this command in poky.yaml.in like 
it used to be?

>> +   $ locale --all-locales | grep en_US.utf8
>> +
>> +If this is not the case, you can reconfigure the ``locales`` package to add it::
>> +
>> +   $ sudo dpkg-reconfigure locales
>> +

How does this actually generate en_US.utf8 locale if we don't explicitly 
add it somewhere? Are you sure it's not because your system has LANG or 
LC_ALL set to that locale and then calling reconfigure it regenerate it?

Also on a side note we tell people to check if "en_US.UTF-8" locale is 
enabled by grepping for "en_US.utf8", is that expected? Why aren't they 
the same?

Cheers,
Quentin

Re: [docs] [PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Antonin Godard]

Hi Guénaël, Quentin,

On Tue Nov 19, 2024 at 10:28 AM CET, Quentin Schulz via lists.yoctoproject.org wrote:
[...]
>>> +You also need to ensure you have the locale ``en_US.UTF-8`` enabled::
>>> +
>
> I believe "the ``en_US.UTF-8`` locale" is more proper here.
>
> On a side note, what's the actual cost of reconfiguring the locales 
> regardless of the presence of en_US.UTF-8?
>
> My point being we could probably have this command in poky.yaml.in like 
> it used to be?
>
>>> +   $ locale --all-locales | grep en_US.utf8
>>> +
>>> +If this is not the case, you can reconfigure the ``locales`` package to add it::
>>> +
>>> +   $ sudo dpkg-reconfigure locales
>>> +
>
> How does this actually generate en_US.utf8 locale if we don't explicitly 
> add it somewhere? Are you sure it's not because your system has LANG or 
> LC_ALL set to that locale and then calling reconfigure it regenerate it?
>
> Also on a side note we tell people to check if "en_US.UTF-8" locale is 
> enabled by grepping for "en_US.utf8", is that expected? Why aren't they 
> the same?

Yes, "locale --all-locales" does return "en_US.utf8" while we install the
"en_US.UTF-8" locale (different string, not sure why).

I seem to be able to build with both Debian/Ubuntu with this command:

  LANG="en_US.UTF-8" locale-gen

Maybe this is a more straightforward approach, closer to what we had before.
What do you think? Guénaël would you be able to update your patch with this
command if this works for you too?


Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Guénaël MULLER]

[-- Attachment #1: Type: text/plain, Size: 6165 bytes --]

Hi Quentin, Antonin,

On Tue, Nov 19, 2024 at 10:28 AM Quentin Schulz <quentin.schulz@cherry.de>
wrote:

> Hi Anotnin, Guénaël,
>
> On 11/18/24 12:00 PM, Antonin Godard via lists.yoctoproject.org wrote:
> > Hi Guénaël,
> >
> > On Mon Nov 18, 2024 at 11:45 AM CET, Guénaël Muller wrote:
> >> Modify locale installation method to be standard accross all
> debian-based distributions.
> >> Pre-existing method is available only on Ubuntu, locale-gen tool has no
> parameter in Debian.
> >>
> >> Signed-off-by: Guénaël Muller <guenael.muller@smile.fr>
> >> Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
> >> Tested-by: Antonin Godard <antonin.godard@bootlin.com>
> >> ---
> >>   documentation/poky.yaml.in                       | 3 +--
> >>   documentation/ref-manual/system-requirements.rst | 8 ++++++++
> >>   2 files changed, 9 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
> >> index c770318f2..66f758fe8 100644
> >> --- a/documentation/poky.yaml.in
> >> +++ b/documentation/poky.yaml.in
> >> @@ -13,8 +13,7 @@ YOCTO_RELEASE_DL_URL :
> "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
> >>   UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip
> texinfo gcc \
> >>        build-essential chrpath socat cpio python3 python3-pip
> python3-pexpect \
> >>        xz-utils debianutils iputils-ping python3-git python3-jinja2 \
> >> -     python3-subunit zstd liblz4-tool file locales libacl1
> >> -     \n\   $ sudo locale-gen en_US.UTF-8"
> >> +     python3-subunit zstd liblz4-tool file locales libacl1"
> >>   FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip
> python3 unzip perl patch \
> >>        diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo
> chrpath \
> >>        ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue
> perl-bignum socat \
> >> diff --git a/documentation/ref-manual/system-requirements.rst
> b/documentation/ref-manual/system-requirements.rst
> >> index 0fc92550a..b33518c7f 100644
> >> --- a/documentation/ref-manual/system-requirements.rst
> >> +++ b/documentation/ref-manual/system-requirements.rst
> >> @@ -152,6 +152,14 @@ with a supported Ubuntu or Debian Linux
> distribution::
> >>
> >>      $ sudo apt install &UBUNTU_HOST_PACKAGES_ESSENTIAL;
> >>
> >> +You also need to ensure you have the locale ``en_US.UTF-8`` enabled::
> >> +
>
> I believe "the ``en_US.UTF-8`` locale" is more proper here.
>

You're Right.


>
> On a side note, what's the actual cost of reconfiguring the locales
> regardless of the presence of en_US.UTF-8?
>

It may take some time depending on how many locales are enabled. The
command is build to generate all enabled locales, it should be technically
possible to generate only one (it's a bash script) but as far I know there
is no such thing in the Debian package from Debian 12.


> My point being we could probably have this command in poky.yaml.in like
> it used to be?
>

> >> +   $ locale --all-locales | grep en_US.utf8
> >> +
> >> +If this is not the case, you can reconfigure the ``locales`` package
> to add it::
> >> +
> >> +   $ sudo dpkg-reconfigure locales
> >> +
>
> How does this actually generate en_US.utf8 locale if we don't explicitly
> add it somewhere? Are you sure it's not because your system has LANG or
> LC_ALL set to that locale and then calling reconfigure it regenerate it?
>
> This runs an interactive program where you have to select explicitly that
you want to add this locale, there is no magic.
The other solution to do similar thing is modifying `/etc/locale.gen` which
seems not more straightforward for me.
I tried it on both podman (container) and classic environnement and it
works correctly.

Also on a side note we tell people to check if "en_US.UTF-8" locale is
> enabled by grepping for "en_US.utf8", is that expected? Why aren't they
> the same?
>
> Cheers,
> Quentin
>


On Wed, Nov 20, 2024 at 5:43 PM Antonin Godard <antonin.godard@bootlin.com>
wrote:

> Hi Guénaël, Quentin,
>
> On Tue Nov 19, 2024 at 10:28 AM CET, Quentin Schulz via
> lists.yoctoproject.org wrote:
> [...]
> >>> +You also need to ensure you have the locale ``en_US.UTF-8`` enabled::
> >>> +
> >
> > I believe "the ``en_US.UTF-8`` locale" is more proper here.
> >
> > On a side note, what's the actual cost of reconfiguring the locales
> > regardless of the presence of en_US.UTF-8?
> >
> > My point being we could probably have this command in poky.yaml.in like
> > it used to be?
> >
> >>> +   $ locale --all-locales | grep en_US.utf8
> >>> +
> >>> +If this is not the case, you can reconfigure the ``locales`` package
> to add it::
> >>> +
> >>> +   $ sudo dpkg-reconfigure locales
> >>> +
> >
> > How does this actually generate en_US.utf8 locale if we don't explicitly
> > add it somewhere? Are you sure it's not because your system has LANG or
> > LC_ALL set to that locale and then calling reconfigure it regenerate it?
> >
> > Also on a side note we tell people to check if "en_US.UTF-8" locale is
> > enabled by grepping for "en_US.utf8", is that expected? Why aren't they
> > the same?
>
> Yes, "locale --all-locales" does return "en_US.utf8" while we install the
> "en_US.UTF-8" locale (different string, not sure why).
>
> I seem to be able to build with both Debian/Ubuntu with this command:
>
>   LANG="en_US.UTF-8" locale-gen
>
> Maybe this is a more straightforward approach, closer to what we had
> before.
> What do you think? Guénaël would you be able to update your patch with this
> command if this works for you too?
>
>
How do you achieve this ? I tried this with podman container (should be the
same with docker) and it clearly doesn't work at all:

podman run -it --rm debian:12 /bin/bash
apt update && apt install locales -y
LANG="en_US.UTF-8" locale-gen

This generates zero locale whereas "dpkg-reconfigure locales" work as
expected.



>
> Antonin
>
> --
> Antonin Godard, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com



Guénaël

[-- Attachment #2: Type: text/html, Size: 9552 bytes --]

Re: [docs] [PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Antonin Godard]

Hi Guénaël,

On Fri Nov 22, 2024 at 10:25 AM CET, Guénaël MULLER wrote:
[...]
>> Yes, "locale --all-locales" does return "en_US.utf8" while we install the
>> "en_US.UTF-8" locale (different string, not sure why).
>>
>> I seem to be able to build with both Debian/Ubuntu with this command:
>>
>>   LANG="en_US.UTF-8" locale-gen
>>
>> Maybe this is a more straightforward approach, closer to what we had
>> before.
>> What do you think? Guénaël would you be able to update your patch with this
>> command if this works for you too?
>>
>>
> How do you achieve this ? I tried this with podman container (should be the
> same with docker) and it clearly doesn't work at all:
>
> podman run -it --rm debian:12 /bin/bash
> apt update && apt install locales -y
> LANG="en_US.UTF-8" locale-gen
>
> This generates zero locale whereas "dpkg-reconfigure locales" work as
> expected.

I may have gotten the impression that this worked, but perhaps not then. The
locales were probably already generated or something... Let's keep it your way
then, and if we come up with another solution we'll update this command.

Thanks for taking the time to look into it!


Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Quentin Schulz]

Hi Guénaël,

On 11/22/24 10:25 AM, Guénaël MULLER wrote:
> Hi Quentin, Antonin,
> 
> On Tue, Nov 19, 2024 at 10:28 AM Quentin Schulz <quentin.schulz@cherry.de>
> wrote:
> 
>> Hi Anotnin, Guénaël,
>>
>> On 11/18/24 12:00 PM, Antonin Godard via lists.yoctoproject.org wrote:
>>> Hi Guénaël,
>>>
>>> On Mon Nov 18, 2024 at 11:45 AM CET, Guénaël Muller wrote:
>>>> Modify locale installation method to be standard accross all
>> debian-based distributions.
>>>> Pre-existing method is available only on Ubuntu, locale-gen tool has no
>> parameter in Debian.
>>>>
>>>> Signed-off-by: Guénaël Muller <guenael.muller@smile.fr>
>>>> Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
>>>> Tested-by: Antonin Godard <antonin.godard@bootlin.com>
>>>> ---
>>>>    documentation/poky.yaml.in                       | 3 +--
>>>>    documentation/ref-manual/system-requirements.rst | 8 ++++++++
>>>>    2 files changed, 9 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
>>>> index c770318f2..66f758fe8 100644
>>>> --- a/documentation/poky.yaml.in
>>>> +++ b/documentation/poky.yaml.in
>>>> @@ -13,8 +13,7 @@ YOCTO_RELEASE_DL_URL :
>> "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
>>>>    UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip
>> texinfo gcc \
>>>>         build-essential chrpath socat cpio python3 python3-pip
>> python3-pexpect \
>>>>         xz-utils debianutils iputils-ping python3-git python3-jinja2 \
>>>> -     python3-subunit zstd liblz4-tool file locales libacl1
>>>> -     \n\   $ sudo locale-gen en_US.UTF-8"
>>>> +     python3-subunit zstd liblz4-tool file locales libacl1"
>>>>    FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip
>> python3 unzip perl patch \
>>>>         diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo
>> chrpath \
>>>>         ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue
>> perl-bignum socat \
>>>> diff --git a/documentation/ref-manual/system-requirements.rst
>> b/documentation/ref-manual/system-requirements.rst
>>>> index 0fc92550a..b33518c7f 100644
>>>> --- a/documentation/ref-manual/system-requirements.rst
>>>> +++ b/documentation/ref-manual/system-requirements.rst
>>>> @@ -152,6 +152,14 @@ with a supported Ubuntu or Debian Linux
>> distribution::
>>>>
>>>>       $ sudo apt install &UBUNTU_HOST_PACKAGES_ESSENTIAL;
>>>>
>>>> +You also need to ensure you have the locale ``en_US.UTF-8`` enabled::
>>>> +
>>
>> I believe "the ``en_US.UTF-8`` locale" is more proper here.
>>
> 
> You're Right.
> 
> 
>>
>> On a side note, what's the actual cost of reconfiguring the locales
>> regardless of the presence of en_US.UTF-8?
>>
> 
> It may take some time depending on how many locales are enabled. The
> command is build to generate all enabled locales, it should be technically
> possible to generate only one (it's a bash script) but as far I know there
> is no such thing in the Debian package from Debian 12.
> 
> 
>> My point being we could probably have this command in poky.yaml.in like
>> it used to be?
>>
> 
>>>> +   $ locale --all-locales | grep en_US.utf8
>>>> +
>>>> +If this is not the case, you can reconfigure the ``locales`` package
>> to add it::
>>>> +
>>>> +   $ sudo dpkg-reconfigure locales
>>>> +
>>
>> How does this actually generate en_US.utf8 locale if we don't explicitly
>> add it somewhere? Are you sure it's not because your system has LANG or
>> LC_ALL set to that locale and then calling reconfigure it regenerate it?
>>
>> This runs an interactive program where you have to select explicitly that
> you want to add this locale, there is no magic.
> The other solution to do similar thing is modifying `/etc/locale.gen` which
> seems not more straightforward for me.
> I tried it on both podman (container) and classic environnement and it
> works correctly.
> 

I assume this works correctly if one uses an interactive container. E.g. 
I'm expecting this won't work with these instructions in a 
Containerfile/Dockerfile for example.

podman run --userns=keep-id --user root debian:bookworm sh -c 'apt-get 
update && apt-get install -y locales; locale --all-locales | grep 
en_US.utf8; dpkg-reconfigure locales; locale --all-locales | grep 
en_US.utf8;'

it'll fail when running dpkg-reconfigure. If you add --frontend 
noninteractive, then it just does nothing.

I'm not sure if we really need noninteractive instructions, but I would 
at least mention that we know it doesn't work for noninteractive 
terminals in a note below the instructions? I understand we may not want 
to instruct users to modify their locales with a simple sed command 
which is apparently the appropriate way?

Cheers,
Quentin

Re: [docs] [PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Guénaël MULLER]

[-- Attachment #1: Type: text/plain, Size: 5754 bytes --]

Hi Antonin, Hi Quentin.

On Mon, Nov 25, 2024 at 11:19 AM Quentin Schulz <quentin.schulz@cherry.de>
wrote:

> Hi Guénaël,
>
> On 11/22/24 10:25 AM, Guénaël MULLER wrote:
> > Hi Quentin, Antonin,
> >
> > On Tue, Nov 19, 2024 at 10:28 AM Quentin Schulz <
> quentin.schulz@cherry.de>
> > wrote:
> >
> >> Hi Anotnin, Guénaël,
> >>
> >> On 11/18/24 12:00 PM, Antonin Godard via lists.yoctoproject.org wrote:
> >>> Hi Guénaël,
> >>>
> >>> On Mon Nov 18, 2024 at 11:45 AM CET, Guénaël Muller wrote:
> >>>> Modify locale installation method to be standard accross all
> >> debian-based distributions.
> >>>> Pre-existing method is available only on Ubuntu, locale-gen tool has
> no
> >> parameter in Debian.
> >>>>
> >>>> Signed-off-by: Guénaël Muller <guenael.muller@smile.fr>
> >>>> Reviewed-by: Yoann Congal <yoann.congal@smile.fr>
> >>>> Tested-by: Antonin Godard <antonin.godard@bootlin.com>
> >>>> ---
> >>>>    documentation/poky.yaml.in                       | 3 +--
> >>>>    documentation/ref-manual/system-requirements.rst | 8 ++++++++
> >>>>    2 files changed, 9 insertions(+), 2 deletions(-)
> >>>>
> >>>> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
> >>>> index c770318f2..66f758fe8 100644
> >>>> --- a/documentation/poky.yaml.in
> >>>> +++ b/documentation/poky.yaml.in
> >>>> @@ -13,8 +13,7 @@ YOCTO_RELEASE_DL_URL :
> >> "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
> >>>>    UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip
> >> texinfo gcc \
> >>>>         build-essential chrpath socat cpio python3 python3-pip
> >> python3-pexpect \
> >>>>         xz-utils debianutils iputils-ping python3-git python3-jinja2 \
> >>>> -     python3-subunit zstd liblz4-tool file locales libacl1
> >>>> -     \n\   $ sudo locale-gen en_US.UTF-8"
> >>>> +     python3-subunit zstd liblz4-tool file locales libacl1"
> >>>>    FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip
> >> python3 unzip perl patch \
> >>>>         diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo
> >> chrpath \
> >>>>         ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue
> >> perl-bignum socat \
> >>>> diff --git a/documentation/ref-manual/system-requirements.rst
> >> b/documentation/ref-manual/system-requirements.rst
> >>>> index 0fc92550a..b33518c7f 100644
> >>>> --- a/documentation/ref-manual/system-requirements.rst
> >>>> +++ b/documentation/ref-manual/system-requirements.rst
> >>>> @@ -152,6 +152,14 @@ with a supported Ubuntu or Debian Linux
> >> distribution::
> >>>>
> >>>>       $ sudo apt install &UBUNTU_HOST_PACKAGES_ESSENTIAL;
> >>>>
> >>>> +You also need to ensure you have the locale ``en_US.UTF-8`` enabled::
> >>>> +
> >>
> >> I believe "the ``en_US.UTF-8`` locale" is more proper here.
> >>
> >
> > You're Right.
> >
> >
> >>
> >> On a side note, what's the actual cost of reconfiguring the locales
> >> regardless of the presence of en_US.UTF-8?
> >>
> >
> > It may take some time depending on how many locales are enabled. The
> > command is build to generate all enabled locales, it should be
> technically
> > possible to generate only one (it's a bash script) but as far I know
> there
> > is no such thing in the Debian package from Debian 12.
> >
> >
> >> My point being we could probably have this command in poky.yaml.in like
> >> it used to be?
> >>
> >
> >>>> +   $ locale --all-locales | grep en_US.utf8
> >>>> +
> >>>> +If this is not the case, you can reconfigure the ``locales`` package
> >> to add it::
> >>>> +
> >>>> +   $ sudo dpkg-reconfigure locales
> >>>> +
> >>
> >> How does this actually generate en_US.utf8 locale if we don't explicitly
> >> add it somewhere? Are you sure it's not because your system has LANG or
> >> LC_ALL set to that locale and then calling reconfigure it regenerate it?
> >>
> >> This runs an interactive program where you have to select explicitly
> that
> > you want to add this locale, there is no magic.
> > The other solution to do similar thing is modifying `/etc/locale.gen`
> which
> > seems not more straightforward for me.
> > I tried it on both podman (container) and classic environnement and it
> > works correctly.
> >
>
> I assume this works correctly if one uses an interactive container. E.g.
> I'm expecting this won't work with these instructions in a
> Containerfile/Dockerfile for example.
>
> podman run --userns=keep-id --user root debian:bookworm sh -c 'apt-get
> update && apt-get install -y locales; locale --all-locales | grep
> en_US.utf8; dpkg-reconfigure locales; locale --all-locales | grep
> en_US.utf8;'
>
> it'll fail when running dpkg-reconfigure. If you add --frontend
> noninteractive, then it just does nothing.
>
> I'm not sure if we really need noninteractive instructions, but I would
> at least mention that we know it doesn't work for noninteractive
> terminals in a note below the instructions? I understand we may not want
> to instruct users to modify their locales with a simple sed command
> which is apparently the appropriate way?
>

I see 2 solutions to solve this issue there:

   - just add "(interactive mode only)"  to the line before the command
   line.
   - add a complete note that explain all the stuff, something like:

 - If your are not using the interactive mode, ``dpkg-reconfigure`` will
not work as expected. To add the locale
you will need to edit ``/etc/locale.gen`` file. A naive way to do this as
root is::

$ echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
$ locale-gen

As soon as we agree on this, I can send a v3.
Antonin, What's your opinion on this ?

>
> Cheers,
> Quentin
>

Guénaël

[-- Attachment #2: Type: text/html, Size: 8589 bytes --]

Re: [docs] [PATCH v2] ref-manual: use standardized method accross both ubuntu and debian for locale install

[Antonin Godard]

Hi Guénaël,

On Mon Nov 25, 2024 at 3:39 PM CET, Guénaël MULLER wrote:
[...]
>> I assume this works correctly if one uses an interactive container. E.g.
>> I'm expecting this won't work with these instructions in a
>> Containerfile/Dockerfile for example.
>>
>> podman run --userns=keep-id --user root debian:bookworm sh -c 'apt-get
>> update && apt-get install -y locales; locale --all-locales | grep
>> en_US.utf8; dpkg-reconfigure locales; locale --all-locales | grep
>> en_US.utf8;'
>>
>> it'll fail when running dpkg-reconfigure. If you add --frontend
>> noninteractive, then it just does nothing.
>>
>> I'm not sure if we really need noninteractive instructions, but I would
>> at least mention that we know it doesn't work for noninteractive
>> terminals in a note below the instructions? I understand we may not want
>> to instruct users to modify their locales with a simple sed command
>> which is apparently the appropriate way?
>>
>
> I see 2 solutions to solve this issue there:
>
>    - just add "(interactive mode only)"  to the line before the command
>    line.
>    - add a complete note that explain all the stuff, something like:

I'd go with this approach (explain it all).

>
>  - If your are not using the interactive mode, ``dpkg-reconfigure`` will
> not work as expected. To add the locale
> you will need to edit ``/etc/locale.gen`` file. A naive way to do this as
> root is::
>
> $ echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
> $ locale-gen

Yes, I would add this after the dpkg-reconfigure method (although maybe I
wouldn't echo into /etc/locale.gen, but rather instruct that "en_US.UTF-8" must
be present and uncommented in this file).

Also, before that, I would simply mention that the dpkg-reconfigure method is
interactive (followed by the /etc/locale.gen method above).


Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com