[RFC] Allow to limit generated binary locales to predefined list

[RFC] Allow to limit generated binary locales to predefined list

[Paul Sokolovsky]

Hello openembedded-devel,

  Binary locale generation for the entire locate set provided by GLIBC
takes noticeable amount of time (several hours). While OE allows to skip
binary locale generation at all, that puts development environment
too far away from "release" one.

  The solution would be to allow to optionally constrain set of
generated locales to a predefined list. I had adhoc patch to do that
for few months now, and several people expressed interest in it. So, I
finally took time to clean it up and make list configurable via OE
variable, and submit it: http://bugs.openembedded.org/show_bug.cgi?id=1966

  List is specified using GLIBC_GENERATE_LOCALES variable, which can
be set e.g. in local.conf:

GLIBC_GENERATE_LOCALES = "en_GB,UTF-8 de_DE,UTF-8"

  Format is <locale>,<encoding> pairs separated by space. Format is
actually based on "SUPPORTED" file generated/provided with GLIBC. For
development purposes, encoding can be just "UTF-8", and for Angstrom,
only en_GB locales is required now. Previous default from OZ/Familiar
was "en_GB,UTF-8 de_DE,UTF-8 fr_FR,UTF-8".


  I'd appreciate comments and testing of the patch above.
  

-- 
Best regards,
 Paul                          mailto:pmiscml@gmail.com

Re: [RFC] Allow to limit generated binary locales to predefined list

[Jan Janssens]

> I finally took time to clean it up and make list configurable via OE
> variable, and submit it: http://bugs.openembedded.org/show_bug.cgi?id=1966
Cool!

> GLIBC_GENERATE_LOCALES = "en_GB,UTF-8 de_DE,UTF-8"
>
>  Format is <locale>,<encoding> pairs separated by space. Format is
> actually based on "SUPPORTED" file generated/provided with GLIBC. For
> development purposes, encoding can be just "UTF-8", and for Angstrom,
> only en_GB locales is required now. Previous default from OZ/Familiar
> was "en_GB,UTF-8 de_DE,UTF-8 fr_FR,UTF-8".
Any special reason to use a comma between the locale and the encoding,
instead of a dot? Not only does the list (at first glance) now look
comma-seperated, almost everywhere a '.' (dot) is used to glue the
encoding to the locale...

Maybe we could even take this one step further and use IMAGE_LINGUAS
and introduce an IMAGE_ENCODINGS and combine these together to get a
list of binary locales to generate?

Jan.

Re: [RFC] Allow to limit generated binary locales to predefined list

[Paul Sokolovsky]

Hello Jan,

Sunday, March 11, 2007, 5:56:28 PM, you wrote:

>> I finally took time to clean it up and make list configurable via OE
>> variable, and submit it: http://bugs.openembedded.org/show_bug.cgi?id=1966
> Cool!

>> GLIBC_GENERATE_LOCALES = "en_GB,UTF-8 de_DE,UTF-8"
>>
>>  Format is <locale>,<encoding> pairs separated by space. Format is
>> actually based on "SUPPORTED" file generated/provided with GLIBC. For
>> development purposes, encoding can be just "UTF-8", and for Angstrom,
>> only en_GB locales is required now. Previous default from OZ/Familiar
>> was "en_GB,UTF-8 de_DE,UTF-8 fr_FR,UTF-8".

> Any special reason to use a comma between the locale and the encoding,
> instead of a dot? Not only does the list (at first glance) now look
> comma-seperated, almost everywhere a '.' (dot) is used to glue the
> encoding to the locale...

  My intention was to do as little (pre/post)processing/interpretation
for GLIBC's SUPPORTED file as possible. Looking at what it has:

aa_DJ.UTF-8 UTF-8
aa_DJ ISO-8859-1

  It's clear why dot couldn't be used - it already appears in
SUPPORT's syntax. So, again, I don't try to interpret it in any way,
just provide means to specify expressions of that syntax using OE variable,
using simple substitution of newlines with spaces, and spaces with
commas.

  Colon or semicolon would be other obvious choices.

> Maybe we could even take this one step further and use IMAGE_LINGUAS
> and introduce an IMAGE_ENCODINGS and combine these together to get a
> list of binary locales to generate?

  Yes, that would be natural extension of that idea. But my patch
doesn't go that far, just offering way to constrain *GLIBC*'s locate
generation.

> Jan.



-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] Allow to limit generated binary locales to predefined list

[Jan Janssens]

>   My intention was to do as little (pre/post)processing/interpretation
> for GLIBC's SUPPORTED file as possible. Looking at what it has:
>
> aa_DJ.UTF-8 UTF-8
> aa_DJ ISO-8859-1
>
>   It's clear why dot couldn't be used - it already appears in
> SUPPORT's syntax. So, again, I don't try to interpret it in any way,
> just provide means to specify expressions of that syntax using OE variable,
> using simple substitution of newlines with spaces, and spaces with
> commas.
Ok, that makes sense.

I don't really get why the SUPPORTED file postfixes all UTF-8 locales
with ".UTF-8" as well as adds " UTF-8". That seems redundant
information; it seems all UTF-8 locales are in the form of
<locale>.UTF-8 UTF-8, while all other encodings simply use <locale>
<encoding>... Anyway, since we don't control that syntax, let's indeed
keep it simple and compatible.

I would like usage of the colon better than the comma though...

Jan.

Re: [RFC] Allow to limit generated binary locales to predefined list

[Paul Sokolovsky]

Hello Jan,

Tuesday, March 13, 2007, 12:53:36 PM, you wrote:

>>   My intention was to do as little (pre/post)processing/interpretation
>> for GLIBC's SUPPORTED file as possible. Looking at what it has:
>>
>> aa_DJ.UTF-8 UTF-8
>> aa_DJ ISO-8859-1
>>
>>   It's clear why dot couldn't be used - it already appears in
>> SUPPORT's syntax. So, again, I don't try to interpret it in any way,
>> just provide means to specify expressions of that syntax using OE variable,
>> using simple substitution of newlines with spaces, and spaces with
>> commas.
> Ok, that makes sense.

> I don't really get why the SUPPORTED file postfixes all UTF-8 locales
> with ".UTF-8" as well as adds " UTF-8". That seems redundant
> information; it seems all UTF-8 locales are in the form of
> <locale>.UTF-8 UTF-8, while all other encodings simply use <locale>
> <encoding>... Anyway, since we don't control that syntax, let's indeed
> keep it simple and compatible.

  And I, after more consideration, that you of course was right, and
we should provide user-visible settings in well-known format, not
something internal. So, I (once again) checked that
glibc-package.bbclass of SUPPORTED's "a[.b] c" syntax just throws away
b, and uses a.c as designator. So, patch is updated:
http://bugs.openembedded.org/show_bug.cgi?id=1966

  Thanks for suggestion!

> I would like usage of the colon better than the comma though...

> Jan.


-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] Allow to limit generated binary locales to predefined list

[Paul Sokolovsky]

Hello,

>  Binary locale generation for the entire locate set provided by GLIBC
> takes noticeable amount of time (several hours).
[...]
>   The solution would be to allow to optionally constrain set of
> generated locales to a predefined list.

> GLIBC_GENERATE_LOCALES = "en_GB.UTF-8 de_DE.UTF-8"

  Committed.


[]

-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] Allow to limit generated binary locales topredefined list

[Mark Gollahon]

[-- Attachment #1: Type: text/plain, Size: 2525 bytes --]

Paul:

This doesn't seem to be working for me.  I've tried:

GLIBC_GENERATE_LOCALES = "en_US en_US.UTF-8"

GLIBC_GENERATE_LOCALES = "en_US,ISO-8859-1 en_US.UTF-8,UTF-8"

GLIBC_GENERATE_LOCALES = "en_US ISO-8859-1,en_US.UTF-8 UTF-8"

in my local.conf file and it gives me this error:

ERROR: Error in executing: /OE/org.openembedded.dev/packages/glibc/glibc_2.5.bb
ERROR: Exception:exceptions.ValueError Message:need more than 1 value to unpack
ERROR: Printing the environment of the function
ERROR:  0057:   # Collate the locales by base and encoding
ERROR:  0058:   encodings = {}
ERROR:  0059:   for l in supported:
ERROR:  0060:           l = l[:-1]
ERROR:  0061:           (locale, charset) = l.split(" ")
ERROR:  0062:           m = dot_re.match(locale)
ERROR:  0063:           if m:
ERROR:  0064:                   locale = m.group(1)
ERROR:  0065:           if not encodings.has_key(locale):
ERROR: Error in executing: /OE/org.openembedded.dev/packages/glibc/glibc_2.5.bb
ERROR: Exception:exceptions.ValueError Message:need more than 1 value to unpack
ERROR: Printing the environment of the function
ERROR:  0057:   if (bb.data.getVar('INHIBIT_PACKAGE_STRIP', d, 1) != '1'):
ERROR:  0058:           stripfunc = ""
ERROR:  0059:           for root, dirs, files in os.walk(dvar):
ERROR:  0060:                   for f in files:
ERROR:  0061:                           file = os.path.join(root, f)
ERROR:  0062:                           if not os.path.islink(file) and not os.path.isdir(file) and isexec(file):
ERROR:  0063:                                   stripfunc += "\trunstrip %s || st=1\n" % (file)
ERROR:  0064:           if not stripfunc == "":
ERROR:  0065:                   from bb import build

Do I need to use bitbake 1.8.x to get this to work??

Regards,
-Mark Gollahon


Paul Sokolovsky wrote ..
> Hello,
> 
> >  Binary locale generation for the entire locate set provided by GLIBC
> > takes noticeable amount of time (several hours).
> [...]
> >   The solution would be to allow to optionally constrain set of
> > generated locales to a predefined list.
> 
> > GLIBC_GENERATE_LOCALES = "en_GB.UTF-8 de_DE.UTF-8"
> 
>   Committed.
> 
> 
> []
> 
> -- 
> Best regards,
>  Paul                            mailto:pmiscml@gmail.com
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: [RFC] Allow to limit generated binary locales topredefined list

[Paul Sokolovsky]

Hello Mark,

Tuesday, March 27, 2007, 12:12:00 AM, you wrote:

> Paul:

> This doesn't seem to be working for me.  I've tried:

> GLIBC_GENERATE_LOCALES = "en_US en_US.UTF-8"

> GLIBC_GENERATE_LOCALES = "en_US,ISO-8859-1 en_US.UTF-8,UTF-8"

> GLIBC_GENERATE_LOCALES = "en_US ISO-8859-1,en_US.UTF-8 UTF-8"

[]

> Do I need to use bitbake 1.8.x to get this to work??

  No, just to read docs/description for things you want to use
carefully. In this respect, I made sure that it can be read in
several places:

1. In this thread, and even in message you quoted below.
2. In the original ticket, #1966.
3. In the commit message.
4. And of course in conf/local.conf.sample in OE tree.


> Regards,
> -Mark Gollahon


> Paul Sokolovsky wrote ..
>> Hello,
>> 
>> >  Binary locale generation for the entire locate set provided by GLIBC
>> > takes noticeable amount of time (several hours).
>> [...]
>> >   The solution would be to allow to optionally constrain set of
>> > generated locales to a predefined list.
>> 
>> > GLIBC_GENERATE_LOCALES = "en_GB.UTF-8 de_DE.UTF-8"
>> 
>>   Committed.
>> 
>> 
>> []
>> 
>> -- 
>> Best regards,
>>  Paul                            mailto:pmiscml@gmail.com
>> 
>> 
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel



-- 
Best regards,
 Paul                            mailto:pmiscml@gmail.com

Re: [RFC] Allow to limit generated binary locales topredefined list

[Florian Boor]

Hi,

Paul Sokolovsky schrieb:
>   No, just to read docs/description for things you want to use
> carefully. In this respect, I made sure that it can be read in
> several places:

wouldn't it be nicer for our users to provide a tiny little hint about what to
search for instead of listing multiple places where to search for it? :-)

Greetings

Florian

-- 
The dream of yesterday                  Florian Boor
is the hope of today                    Tel: +49 271-771091-15
and the reality of tomorrow.            Fax: +49 271-771091-19
[Robert Hutchings Goddard, 1904]        florian.boor@kernelconcepts.de

1D78 2D4D 6C53 1CA4 5588  D07B A8E7 940C 25B7 9A76

Re: [RFC] Allow to limit generated binary locales to predefined list

[Rolf Leggewie]

Paul Sokolovsky wrote:
>   Binary locale generation for the entire locate set provided by GLIBC
> takes noticeable amount of time (several hours).
[...]
>   The solution would be to allow to optionally constrain set of
> generated locales to a predefined list.

Good idea.  And I can say that the patch to glibc-package.bbclass I 
prepared with Paul's help (big thank you!) has been working fine for me 
for a few weeks now.

+1 from me.