[RFC] locales

[RFC] locales

[Sergey Lapin]

Hi, all!

Note: I could be fundamentally wrong here, so, please feel free to
point me to right direction in this case.

As it was discussed on #oe, locale infrastructure lacks implementation
on oe.dev, and, in particular, in Angstrom.

With a bit of research I done on this subject made me to come up with
a few ideas, which I'd like to know your opinion about.

Present status:
*-locale* packages are generated, but not put on image. These packages
contain message translations.
* Only glibc locale which is put on image (if any) is en_GB.

Infrastructure problem:
* We need a way to set up automatic locale package installation during
image build
according to some subset of languages/locales.
* We need a way to install "language" in simple user-friendly way. I
mean here not only locale packages, but also various
language-dependent files (docs, keymap settings, various configuration
files, etc.).

Various random problems encountered:
* GPE locale.alias for gpe-dm needs cleanup (get rid of non-UTF8 locales?)
* libx11 locale.alias needs cleanup (setup for UTF8-only system).

So, if we go utf8-only, we need to do a great cleanup/testing job here
to settle things up.

As for infrastructure, I see several questionable methods of solving this:

1. each package RRECOMMENDS its locale packages for locales mentioned
in local.conf variable, and additional ones, which are related.

2. image RRECOMMENDS locale packages blindly for all normal packages
for languages which are in a var mentioned in 1.

3. all -locale packages generated during builds are written to some
special lists, for each language. Then meta-packages are generated
from these lists.

...

So, any ideas?

Re: [RFC] locales

[Holger Freyther]

Am 20.06.2007 um 12:14 schrieb Sergey Lapin:

>
> As for infrastructure, I see several questionable methods of  
> solving this:
>
> 1. each package RRECOMMENDS its locale packages for locales mentioned
> in local.conf variable, and additional ones, which are related.


This would open the PACKAGE_ARCH hell. _armebdeenengbjpkoru to get a  
package
built for ARM EB with DE, EN, EN_GB, JP, KO and RU locales

>
> 2. image RRECOMMENDS locale packages blindly for all normal packages
> for languages which are in a var mentioned in 1.

This will require too much space in most of the cases. I'm thinking  
here of non Glib based
systems.


>
> 3. all -locale packages generated during builds are written to some
> special lists, for each language. Then meta-packages are generated
> from these lists.

This could work but......

the fundamental question is what is OpenSUSE, Fedora, Ubuntu doing to  
select a language? E.g. in the case of Opie you want to have a  
language for Opie, in the case for GPE you want to have gettext files  
for the whole system and GPE. You don't want to install translations  
where you don't have the package installed but when you install the  
package you want to have the translations automatically installed.

I don't think this is in the scope of OE but in the scope of  
distributions working together with the environment (GPE, Opie) to  
find a proper solution. Everything to implement it is already there....


z.

Re: [RFC] locales

[Sergey Lapin]

> > 3. all -locale packages generated during builds are written to some
> > special lists, for each language. Then meta-packages are generated
> > from these lists.
>
> This could work but......
>
> the fundamental question is what is OpenSUSE, Fedora, Ubuntu doing to
IIRC they do not package translations separately.

> select a language? E.g. in the case of Opie you want to have a
> language for Opie, in the case for GPE you want to have gettext files
> for the whole system and GPE. You don't want to install translations
> where you don't have the package installed but when you install the
> package you want to have the translations automatically installed.
>
> I don't think this is in the scope of OE but in the scope of
> distributions working together with the environment (GPE, Opie) to
> find a proper solution. Everything to implement it is already there....
big distros don't require so much space economy, so they package
translations inside packages, so the problem is solved. I personally
like RRECOMMENDS solution, like
reciepe foo generates foo, foo-locale, foo-doc, foo-dev.
we have var IMAGE_LOCALES="en_GB en_US"
then RRECOMMENDS_foo will be = "${PN}-locale-en-gb ${PN}-locale-en-us"

so, like this. I think of it as simplest solution. Actually, if
package have some additional locale-dependant files, they could be
added to their corresponding -locale-x-y-z packages.
as for z component I think like en_US.ISO8859-1 will be -locale-en-us-iso8859-1.

All the best,
S.

Re: [RFC] locales

[Holger Freyther]

Am 20.06.2007 um 16:25 schrieb Sergey Lapin:


> big distros don't require so much space economy, so they package
> translations inside packages, so the problem is solved. I personally
> like RRECOMMENDS solution, like
> reciepe foo generates foo, foo-locale, foo-doc, foo-dev.
> we have var IMAGE_LOCALES="en_GB en_US"
> then RRECOMMENDS_foo will be = "${PN}-locale-en-gb ${PN}-locale-en-us"

No,
see point one (the one you have removed from this mail).


z.

Re: [RFC] locales

[Sergey Lapin]

> No,
> see point one (the one you have removed from this mail).

So as I see, the best variant is near var.3 in my first mail.

so we setup:
IMAGE_LOCALES="... ", which is if unset, means none.
during image build process, all translations packages are built as
usual, but names for them are written in list in tmp dir (for each
language is separate list).
then we put it into variable e.g. IMAGE_LOCALE_PACKAGES according to
setting of IMAGE_LOCALES and hope that image itself will add them

Is this variant o.k.?

S.

Re: [RFC] locales

[Paul Sokolovsky]

Hello Sergey,

Wednesday, June 20, 2007, 5:25:34 PM, you wrote:

[]

>> select a language? E.g. in the case of Opie you want to have a
>> language for Opie, in the case for GPE you want to have gettext files
>> for the whole system and GPE. You don't want to install translations
>> where you don't have the package installed but when you install the
>> package you want to have the translations automatically installed.
>>
>> I don't think this is in the scope of OE but in the scope of
>> distributions working together with the environment (GPE, Opie) to
>> find a proper solution. Everything to implement it is already there....
> big distros don't require so much space economy, so they package
> translations inside packages, so the problem is solved. I personally
> like RRECOMMENDS solution, like
> reciepe foo generates foo, foo-locale, foo-doc, foo-dev.
> we have var IMAGE_LOCALES="en_GB en_US"
> then RRECOMMENDS_foo will be = "${PN}-locale-en-gb ${PN}-locale-en-us"

        This will lead to contamination of package metadata and
consequently to errors, hard to track for many people (for example, if
you change IMAGE_LOCALES later, previously built packages will still
pull old locale set). L19n really needs special handling.



-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] locales

[Michael Krelin]

I do not see why do we want to get rid of non-utf locales. Wouldn't an 
attempt to make utf-locales work more productive? And I believe making 
things work should be of higher priority than making packaging friendlier.

At the moment, we have troubles entering non-latin utf8 in x11. That 
should be fixed before going any further. And before you can fix it you 
need to find out what exactly the problem is. This is where I'd suggest 
that we start at.

Love,
H

Sergey Lapin wrote:
> Hi, all!
> 
> Note: I could be fundamentally wrong here, so, please feel free to
> point me to right direction in this case.
> 
> As it was discussed on #oe, locale infrastructure lacks implementation
> on oe.dev, and, in particular, in Angstrom.
> 
> With a bit of research I done on this subject made me to come up with
> a few ideas, which I'd like to know your opinion about.
> 
> Present status:
> *-locale* packages are generated, but not put on image. These packages
> contain message translations.
> * Only glibc locale which is put on image (if any) is en_GB.
> 
> Infrastructure problem:
> * We need a way to set up automatic locale package installation during
> image build
> according to some subset of languages/locales.
> * We need a way to install "language" in simple user-friendly way. I
> mean here not only locale packages, but also various
> language-dependent files (docs, keymap settings, various configuration
> files, etc.).
> 
> Various random problems encountered:
> * GPE locale.alias for gpe-dm needs cleanup (get rid of non-UTF8 locales?)
> * libx11 locale.alias needs cleanup (setup for UTF8-only system).
> 
> So, if we go utf8-only, we need to do a great cleanup/testing job here
> to settle things up.
> 
> As for infrastructure, I see several questionable methods of solving this:
> 
> 1. each package RRECOMMENDS its locale packages for locales mentioned
> in local.conf variable, and additional ones, which are related.
> 
> 2. image RRECOMMENDS locale packages blindly for all normal packages
> for languages which are in a var mentioned in 1.
> 
> 3. all -locale packages generated during builds are written to some
> special lists, for each language. Then meta-packages are generated
> from these lists.
> 
> ...
> 
> So, any ideas?
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> 

Re: [RFC] locales

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Krelin schreef:
> I do not see why do we want to get rid of non-utf locales. Wouldn't an 
> attempt to make utf-locales work more productive? And I believe making 
> things work should be of higher priority than making packaging friendlier.

These two can be done in parallel, but I agree that making utf work flawlessly would be
easier and more 'visible' to the end user.


regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFGeRE7MkyGM64RGpERAh0+AJwLw5adGDfqpaHrwx1829wDo1pR+gCcDMOI
lUSZuk/fHLd5j/k56yOb1ko=
=ATr7
-----END PGP SIGNATURE-----

Re: [RFC] locales

[Paul Sokolovsky]

Hello Sergey,

Wednesday, June 20, 2007, 1:14:41 PM, you wrote:

> Hi, all!

> Note: I could be fundamentally wrong here, so, please feel free to
> point me to right direction in this case.

> As it was discussed on #oe, locale infrastructure lacks implementation
> on oe.dev, and, in particular, in Angstrom.

        That's true in the sense that there's no special infra to deal
with l19n issues efficiently. Otherwise, basic support on package
management level is there and works well (on that very package management
level).

        Going for elaboration of this support is opening Pandora's box.
There going to be just too many issues, and they will swamp whoever go
for them. At that's at the time, when we have more pressing issues on
all levels (OE - many packages not up-to-date/broken; Angstrom - lots
of clean up to do, and finally to make release; Specific devices -
lotsa kernel and related stuff to support/improve).

        So, I feel like currently it's bad time to go for l19n
problems.

> With a bit of research I done on this subject made me to come up with
> a few ideas, which I'd like to know your opinion about.

> Present status:
> *-locale* packages are generated, but not put on image. These packages
> contain message translations.
> * Only glibc locale which is put on image (if any) is en_GB.

        Yes, as of now, Angstrom is shipped with the default locale
only. Users who need other locale, can easily install it using package
manager (and yes, this easiness can be improved even more).

> Infrastructure problem:
> * We need a way to set up automatic locale package installation during
> image build according to some subset of languages/locales.

        On OE level, it's possible. On Angstrom level, we'd need to
decide which will be that "subset". And one decision was already made
- as locales (as in glibc locale package) are big in size, and there's
no definitive subset of size=N, N>1 which will allow to cover needs of
greater audience than subset of size=N-1, let there be one default
locale, and let users use standard means of customizing the install -
a package manager.

> * We need a way to install "language" in simple user-friendly way. I
> mean here not only locale packages, but also various
> language-dependent files (docs, keymap settings, various configuration
> files, etc.).

> Various random problems encountered:
> * GPE locale.alias for gpe-dm needs cleanup (get rid of non-UTF8 locales?)
> * libx11 locale.alias needs cleanup (setup for UTF8-only system).

> So, if we go utf8-only, we need to do a great cleanup/testing job here
> to settle things up.

> As for infrastructure, I see several questionable methods of solving this:

> 1. each package RRECOMMENDS its locale packages for locales mentioned
> in local.conf variable, and additional ones, which are related.

> 2. image RRECOMMENDS locale packages blindly for all normal packages
> for languages which are in a var mentioned in 1.

        These games with dependencies will do more harm than good.
l19n is pretty specific thing, and requires adhoc handling on another
level.

> 3. all -locale packages generated during builds are written to some
> special lists, for each language. Then meta-packages are generated
> from these lists.

        And this sounds a bit tangled. What such list will give you?
Why it needs to be written while generating packages? Why
"ls deploy/ipk/" is worse than such list?



        Ok, let me just dump my thoughts on how I'd do that:

1. Use ROOTFS_POSTPROCESS_COMMAND to do this stuff, don't patch the
rest of bitbake/OE.
2. ROOTFS_POSTPROCESS_COMMAND is run after rootfs is fully created, in
particular, when all packages are recursively installed. The list of
them is in /usr/lib/ipkg/status.
3. Read the list, and for each package PKG in it, try to install
package PKG-locale-LL. Skip non-existent packages.
4. Voila


> ...

> So, any ideas?

-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] locales

[Sergey Lapin]

Paul Sokolovsky wrote:
> 
>         That's true in the sense that there's no special infra to deal
> with l19n issues efficiently. Otherwise, basic support on package
> management level is there and works well (on that very package management
> level).
> 
>         Going for elaboration of this support is opening Pandora's box.
> There going to be just too many issues, and they will swamp whoever go
> for them. At that's at the time, when we have more pressing issues on
> all levels (OE - many packages not up-to-date/broken; Angstrom - lots
> of clean up to do, and finally to make release; Specific devices -
> lotsa kernel and related stuff to support/improve).
I have a feeling that that will go on forever - there are always
broken packages, not to mention improvements, especially on kernel side.
There's no stopping point, and that's not bad.


> 
>         So, I feel like currently it's bad time to go for l19n
> problems.
It seems that good time will never come, since there are always
other things to be done.

>         Yes, as of now, Angstrom is shipped with the default locale
> only. Users who need other locale, can easily install it using package
> manager (and yes, this easiness can be improved even more).

Any interesting details here, please?

> 
>> Infrastructure problem:
>> * We need a way to set up automatic locale package installation during
>> image build according to some subset of languages/locales.
> 
>         On OE level, it's possible. On Angstrom level, we'd need to
> decide which will be that "subset". And one decision was already made
> - as locales (as in glibc locale package) are big in size, and there's
> no definitive subset of size=N, N>1 which will allow to cover needs of
> greater audience than subset of size=N-1, let there be one default
> locale, and let users use standard means of customizing the install -
> a package manager.
It is not possible for some devices, and too hard on others
(due to lack of device support, for example, and need to provide
powerful showcase).

>         Ok, let me just dump my thoughts on how I'd do that:
> 
> 1. Use ROOTFS_POSTPROCESS_COMMAND to do this stuff, don't patch the
> rest of bitbake/OE.
> 2. ROOTFS_POSTPROCESS_COMMAND is run after rootfs is fully created, in
> particular, when all packages are recursively installed. The list of
> them is in /usr/lib/ipkg/status.
> 3. Read the list, and for each package PKG in it, try to install
> package PKG-locale-LL. Skip non-existent packages.
> 4. Voila

And here you provide solution to build image, not solving
issues for user (he installs application and expects
for package manager to install his locales for him
automatically. So, a bit of patching of ipkg is needed.
Any cons?

Re: [RFC] locales

[Paul Sokolovsky]

Hello Sergey,

Thursday, June 21, 2007, 1:05:19 AM, you wrote:

> Paul Sokolovsky wrote:
>> 
>>         That's true in the sense that there's no special infra to deal
>> with l19n issues efficiently. Otherwise, basic support on package
>> management level is there and works well (on that very package management
>> level).
>> 
>>         Going for elaboration of this support is opening Pandora's box.
>> There going to be just too many issues, and they will swamp whoever go
>> for them. At that's at the time, when we have more pressing issues on
>> all levels (OE - many packages not up-to-date/broken; Angstrom - lots
>> of clean up to do, and finally to make release; Specific devices -
>> lotsa kernel and related stuff to support/improve).
> I have a feeling that that will go on forever - there are always
> broken packages, not to mention improvements, especially on kernel side.
> There's no stopping point, and that's not bad.
>>
>>         So, I feel like currently it's bad time to go for l19n
>> problems.
> It seems that good time will never come, since there are always
> other things to be done.

        Nice bit of philosophy in midst of technical/organization
discussion. But even philosophical theory of changes has notions of
"less appropriate" and "more appropriate" for a given event in a given
point of time. Anyway, that was all my IMHO, and YMMV.

>>         Yes, as of now, Angstrom is shipped with the default locale
>> only. Users who need other locale, can easily install it using package
>> manager (and yes, this easiness can be improved even more).

> Any interesting details here, please?

        My idea is that there should be a tool, frontend to package
manager, allowing to find and install useful things easily for end
user (but not necessarily in optimal way). To not create more entities
than needed, in my dream I see the very package manager reuse for
that, under following scheme:

1. Multiple section support added to ipk's.
2. Package manager provides ability to to filter list by section
(likely already done).
3. A separate section is created, with catchy name like
"Oh-so-cool-goodies".
4. Within that section, there live virtual packages (empty with just
Depends on real packages), within also boring names like "Click here
to install translations for language X" or "Click here to install
bunch'o'games".
5. There's a shortcut on desktop, which starts package manager with
the mentioned section filter applied.

>> 
>>> Infrastructure problem:
>>> * We need a way to set up automatic locale package installation during
>>> image build according to some subset of languages/locales.
>> 
>>         On OE level, it's possible. On Angstrom level, we'd need to
>> decide which will be that "subset". And one decision was already made
>> - as locales (as in glibc locale package) are big in size, and there's
>> no definitive subset of size=N, N>1 which will allow to cover needs of
>> greater audience than subset of size=N-1, let there be one default
>> locale, and let users use standard means of customizing the install -
>> a package manager.
> It is not possible for some devices, and too hard on others
> (due to lack of device support, for example, and need to provide
> powerful showcase).

        Locales, devices, and phase of the moon have no correlation,
sorry. It should be possible to install any locale on any device
regardless of the moon phase.

>>         Ok, let me just dump my thoughts on how I'd do that:
>> 
>> 1. Use ROOTFS_POSTPROCESS_COMMAND to do this stuff, don't patch the
>> rest of bitbake/OE.
>> 2. ROOTFS_POSTPROCESS_COMMAND is run after rootfs is fully created, in
>> particular, when all packages are recursively installed. The list of
>> them is in /usr/lib/ipkg/status.
>> 3. Read the list, and for each package PKG in it, try to install
>> package PKG-locale-LL. Skip non-existent packages.
>> 4. Voila

> And here you provide solution to build image, not solving
> issues for user

        You were already provided idea of the corresponding
solution for runtime use on IRC. Just to have it all in one place,
here what it was:

1. You write an app which scans currently installed list of packages.
2. For each package PKG, it tries to install PKG-locale-LL, skipping
any errors.
3. Users run this update-locales tool after installing new packages.

> (he installs application and expects
> for package manager to install his locales for him
> automatically.
> So, a bit of patching of ipkg is needed.
> Any cons?

        Pretty mad expectations. Lart that user, just in case. But
even that is possible, and of course without (adhoc) ipkg patching. In
the worst case, we can add that update-locales script as post-install
for each package which has locales. But that's again contamination of
metadata. Instead, ipkg should allow global post-install hook, and
update-locales should be there.


-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] locales

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Sokolovsky schreef:


> 1. Multiple section support added to ipk's.

or: 1. Switch to .debs and use debtags
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFGei6VMkyGM64RGpERAiBRAJ46llj7GxLcRBpmQBGU3uY1NKfWfwCfZQC5
6ZGTtzpvS/21JdovcX6Ab/A=
=y4uH
-----END PGP SIGNATURE-----