[U-Boot] [PATCH] fix linker generated lists on non-C locales

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Marek Vasut]

Dear David Marqvar Nielsen,

> Hi Marex,
> 
> Any change that the fix for the problem this patch addresses will make it
> into 2013.01 ?
> I will test the variant "*LC_ALL=  *LC_COLLATE=C sort -u | \
> tomorrow.

Please always reply to the list. I'm Ccing most of the involved people.

Best regards,
Marek Vasut

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Pavel Herrmann]

Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result is a build
that has no available commands. Setting LC_ALL=C for the generator script helps.

Signed-off-by: Pavel Herrmann <morpheus.ibis@gmail.com>
---
 helper.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/helper.mk b/helper.mk
index 79a1da0..2b75182 100644
--- a/helper.mk
+++ b/helper.mk
@@ -35,7 +35,7 @@
 #    line. Next, append "@" at the end and print the line. Finally,
 #    append "~" at the end of line. This will make sense in conjunction
 #    with 6) and 7).
-# 7) Sort the lines. It is imperative to use LC_COLLATE=C here because
+# 7) Sort the lines. It is imperative to use LC_ALL=C here because
 #    with this, the "\a" symbol is first and "~" symbol is last. Any
 #    other symbols fall inbetween. Symbols like "@", which marks the
 #    end of current line (representing current section) and ".", which
@@ -57,7 +57,7 @@ $(1): $(2)
 		-e 's/\.[^\.]\+$$$$//' \
 		-e ':s /^.\+$$$$/ { p;s/^\(.*\)\.[^\.]*$$$$/\1/;b s }' | \
 	sed -n -e 'h;s/$$$$/\a/p;g;s/$$$$/@/p;g;s/$$$$/~/p;' | \
-	LC_COLLATE=C sort -u | \
+	LC_ALL=C sort -u | \
 	sed -n -e '/\a$$$$/ { s/\./_/g;s/\a$$$$/__start = .;/p; }'\
 		-e '/~$$$$/ { s/\./_/g;s/~$$$$/__end = .;/p; }'\
 		-e '/@$$$$/ { s/\(.*\)@$$$$/*(SORT(\1.*));/p }' > $(1)
-- 
1.7.12.4

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Marek Vasut]

Dear Pavel Herrmann,

> Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result is a
> build that has no available commands. Setting LC_ALL=C for the generator
> script helps.
> 
> Signed-off-by: Pavel Herrmann <morpheus.ibis@gmail.com>
> ---
>  helper.mk | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/helper.mk b/helper.mk
> index 79a1da0..2b75182 100644
> --- a/helper.mk
> +++ b/helper.mk
> @@ -35,7 +35,7 @@
>  #    line. Next, append "@" at the end and print the line. Finally,
>  #    append "~" at the end of line. This will make sense in conjunction
>  #    with 6) and 7).
> -# 7) Sort the lines. It is imperative to use LC_COLLATE=C here because
> +# 7) Sort the lines. It is imperative to use LC_ALL=C here because
>  #    with this, the "\a" symbol is first and "~" symbol is last. Any
>  #    other symbols fall inbetween. Symbols like "@", which marks the
>  #    end of current line (representing current section) and ".", which
> @@ -57,7 +57,7 @@ $(1): $(2)
>  		-e 's/\.[^\.]\+$$$$//' \
>  		-e ':s /^.\+$$$$/ { p;s/^\(.*\)\.[^\.]*$$$$/\1/;b s }' | \
>  	sed -n -e 'h;s/$$$$/\a/p;g;s/$$$$/@/p;g;s/$$$$/~/p;' | \
> -	LC_COLLATE=C sort -u | \
> +	LC_ALL=C sort -u | \

How will this work? LC_COLLATE is supposed to adjust the behavior of "sort" 
command, without it, "sort" will not behave as expected.

Best regards,
Marek Vasut

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Pavel Herrmann]

On Sunday 02 of December 2012 18:50:53 Marek Vasut wrote:
> Dear Pavel Herrmann,
> 
> > Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result is a
> > build that has no available commands. Setting LC_ALL=C for the generator
> > script helps.
> > 
> > Signed-off-by: Pavel Herrmann <morpheus.ibis@gmail.com>
> > ---
> > 
> >  helper.mk | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/helper.mk b/helper.mk
> > index 79a1da0..2b75182 100644
> > --- a/helper.mk
> > +++ b/helper.mk
> > @@ -35,7 +35,7 @@
> > 
> >  #    line. Next, append "@" at the end and print the line. Finally,
> >  #    append "~" at the end of line. This will make sense in conjunction
> >  #    with 6) and 7).
> > 
> > -# 7) Sort the lines. It is imperative to use LC_COLLATE=C here because
> > +# 7) Sort the lines. It is imperative to use LC_ALL=C here because
> > 
> >  #    with this, the "\a" symbol is first and "~" symbol is last. Any
> >  #    other symbols fall inbetween. Symbols like "@", which marks the
> >  #    end of current line (representing current section) and ".", which
> > 
> > @@ -57,7 +57,7 @@ $(1): $(2)
> > 
> >  		-e 's/\.[^\.]\+$$$$//' \
> >  		-e ':s /^.\+$$$$/ { p;s/^\(.*\)\.[^\.]*$$$$/\1/;b s }' | \
> >  	
> >  	sed -n -e 'h;s/$$$$/\a/p;g;s/$$$$/@/p;g;s/$$$$/~/p;' | \
> > 
> > -	LC_COLLATE=C sort -u | \
> > +	LC_ALL=C sort -u | \
> 
> How will this work? LC_COLLATE is supposed to adjust the behavior of "sort"
> command, without it, "sort" will not behave as expected.

Clearly, LC_COLLATE is not enough for it to behave correctly, as seen on my 
setup.
This works because LC_ALL sets LC_COLLATE, as well as all other LC_* 
environment variables.

Pavel Herrmann

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Marek Vasut]

Dear Pavel Herrmann,

> On Sunday 02 of December 2012 18:50:53 Marek Vasut wrote:
> > Dear Pavel Herrmann,
> > 
> > > Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result is
> > > a build that has no available commands. Setting LC_ALL=C for the
> > > generator script helps.
> > > 
> > > Signed-off-by: Pavel Herrmann <morpheus.ibis@gmail.com>
> > > ---
> > > 
> > >  helper.mk | 4 ++--
> > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/helper.mk b/helper.mk
> > > index 79a1da0..2b75182 100644
> > > --- a/helper.mk
> > > +++ b/helper.mk
> > > @@ -35,7 +35,7 @@
> > > 
> > >  #    line. Next, append "@" at the end and print the line. Finally,
> > >  #    append "~" at the end of line. This will make sense in
> > >  conjunction #    with 6) and 7).
> > > 
> > > -# 7) Sort the lines. It is imperative to use LC_COLLATE=C here because
> > > +# 7) Sort the lines. It is imperative to use LC_ALL=C here because
> > > 
> > >  #    with this, the "\a" symbol is first and "~" symbol is last. Any
> > >  #    other symbols fall inbetween. Symbols like "@", which marks the
> > >  #    end of current line (representing current section) and ".", which
> > > 
> > > @@ -57,7 +57,7 @@ $(1): $(2)
> > > 
> > >  		-e 's/\.[^\.]\+$$$$//' \
> > >  		-e ':s /^.\+$$$$/ { p;s/^\(.*\)\.[^\.]*$$$$/\1/;b s }' | \
> > >  	
> > >  	sed -n -e 'h;s/$$$$/\a/p;g;s/$$$$/@/p;g;s/$$$$/~/p;' | \
> > > 
> > > -	LC_COLLATE=C sort -u | \
> > > +	LC_ALL=C sort -u | \
> > 
> > How will this work? LC_COLLATE is supposed to adjust the behavior of
> > "sort" command, without it, "sort" will not behave as expected.
> 
> Clearly, LC_COLLATE is not enough for it to behave correctly, as seen on my
> setup.
> This works because LC_ALL sets LC_COLLATE, as well as all other LC_*
> environment variables.

Do you have any reference where I can verify this claim please?

Best regards,
Marek Vasut

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Pavel Herrmann]

On Sunday 02 of December 2012 19:12:17 Marek Vasut wrote:
> Dear Pavel Herrmann,
> 
> > On Sunday 02 of December 2012 18:50:53 Marek Vasut wrote:
> > > Dear Pavel Herrmann,
> > > 
> > > > Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result is
> > > > a build that has no available commands. Setting LC_ALL=C for the
> > > > generator script helps.
> > > > 
> > > > Signed-off-by: Pavel Herrmann <morpheus.ibis@gmail.com>
> > > > ---
> > > > 
> > > >  helper.mk | 4 ++--
> > > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/helper.mk b/helper.mk
> > > > index 79a1da0..2b75182 100644
> > > > --- a/helper.mk
> > > > +++ b/helper.mk
> > > > @@ -35,7 +35,7 @@
> > > > 
> > > >  #    line. Next, append "@" at the end and print the line. Finally,
> > > >  #    append "~" at the end of line. This will make sense in
> > > >  conjunction #    with 6) and 7).
> > > > 
> > > > -# 7) Sort the lines. It is imperative to use LC_COLLATE=C here
> > > > because
> > > > +# 7) Sort the lines. It is imperative to use LC_ALL=C here because
> > > > 
> > > >  #    with this, the "\a" symbol is first and "~" symbol is last. Any
> > > >  #    other symbols fall inbetween. Symbols like "@", which marks the
> > > >  #    end of current line (representing current section) and ".",
> > > >  which
> > > > 
> > > > @@ -57,7 +57,7 @@ $(1): $(2)
> > > > 
> > > >  		-e 's/\.[^\.]\+$$$$//' \
> > > >  		-e ':s /^.\+$$$$/ { p;s/^\(.*\)\.[^\.]*$$$$/\1/;b s }' | \
> > > >  	
> > > >  	sed -n -e 'h;s/$$$$/\a/p;g;s/$$$$/@/p;g;s/$$$$/~/p;' | \
> > > > 
> > > > -	LC_COLLATE=C sort -u | \
> > > > +	LC_ALL=C sort -u | \
> > > 
> > > How will this work? LC_COLLATE is supposed to adjust the behavior of
> > > "sort" command, without it, "sort" will not behave as expected.
> > 
> > Clearly, LC_COLLATE is not enough for it to behave correctly, as seen on
> > my
> > setup.
> > This works because LC_ALL sets LC_COLLATE, as well as all other LC_*
> > environment variables.
> 
> Do you have any reference where I can verify this claim please?

Which claim? that this helps, or that LC_ALL sets LC_COLLATE as well?

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Marek Vasut]

Dear Pavel Herrmann,

> On Sunday 02 of December 2012 19:12:17 Marek Vasut wrote:
> > Dear Pavel Herrmann,
> > 
> > > On Sunday 02 of December 2012 18:50:53 Marek Vasut wrote:
> > > > Dear Pavel Herrmann,
> > > > 
> > > > > Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result
> > > > > is a build that has no available commands. Setting LC_ALL=C for
> > > > > the generator script helps.
> > > > > 
> > > > > Signed-off-by: Pavel Herrmann <morpheus.ibis@gmail.com>
> > > > > ---
> > > > > 
> > > > >  helper.mk | 4 ++--
> > > > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > > > 
> > > > > diff --git a/helper.mk b/helper.mk
> > > > > index 79a1da0..2b75182 100644
> > > > > --- a/helper.mk
> > > > > +++ b/helper.mk
> > > > > @@ -35,7 +35,7 @@
> > > > > 
> > > > >  #    line. Next, append "@" at the end and print the line.
> > > > >  Finally, #    append "~" at the end of line. This will make sense
> > > > >  in conjunction #    with 6) and 7).
> > > > > 
> > > > > -# 7) Sort the lines. It is imperative to use LC_COLLATE=C here
> > > > > because
> > > > > +# 7) Sort the lines. It is imperative to use LC_ALL=C here because
> > > > > 
> > > > >  #    with this, the "\a" symbol is first and "~" symbol is last.
> > > > >  Any #    other symbols fall inbetween. Symbols like "@", which
> > > > >  marks the #    end of current line (representing current section)
> > > > >  and ".", which
> > > > > 
> > > > > @@ -57,7 +57,7 @@ $(1): $(2)
> > > > > 
> > > > >  		-e 's/\.[^\.]\+$$$$//' \
> > > > >  		-e ':s /^.\+$$$$/ { p;s/^\(.*\)\.[^\.]*$$$$/\1/;b s }' | 
\
> > > > >  	
> > > > >  	sed -n -e 'h;s/$$$$/\a/p;g;s/$$$$/@/p;g;s/$$$$/~/p;' | \
> > > > > 
> > > > > -	LC_COLLATE=C sort -u | \
> > > > > +	LC_ALL=C sort -u | \
> > > > 
> > > > How will this work? LC_COLLATE is supposed to adjust the behavior of
> > > > "sort" command, without it, "sort" will not behave as expected.
> > > 
> > > Clearly, LC_COLLATE is not enough for it to behave correctly, as seen
> > > on my
> > > setup.
> > > This works because LC_ALL sets LC_COLLATE, as well as all other LC_*
> > > environment variables.
> > 
> > Do you have any reference where I can verify this claim please?
> 
> Which claim? that this helps, or that LC_ALL sets LC_COLLATE as well?

You claim the following:

"This works because LC_ALL sets LC_COLLATE, as well as all other LC_* 
environment variables."

Where can I verify this? Do you have any link to any documentation?

Best regards,
Marek Vasut

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Wolfgang Denk]

Dear Marek Vasut,

In message <201212022013.13540.marex@denx.de> you wrote:
> 
> "This works because LC_ALL sets LC_COLLATE, as well as all other LC_* 
> environment variables."
> 
> Where can I verify this? Do you have any link to any documentation?

Is it really so difficult to find TFM these days?

Try: man locale

...
LC_ALL If set to a non-empty string value, override the values of all
the other internationalization variables.
...

Is this clear enough?

As mentioned before, if this variable is set, it can usually be
considered a bug.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Life would be so much easier if we could  just  look  at  the  source
code.                                                   -- Dave Olson

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Pavel Herrmann]

On Sunday 02 of December 2012 19:12:17 Marek Vasut wrote:
> Dear Pavel Herrmann,
> 
> > On Sunday 02 of December 2012 18:50:53 Marek Vasut wrote:
> > > Dear Pavel Herrmann,
> > > 
> > > > Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result is
> > > > a build that has no available commands. Setting LC_ALL=C for the
> > > > generator script helps.
> > > > 
> > > > Signed-off-by: Pavel Herrmann <morpheus.ibis@gmail.com>
> > > > ---
> > > > 
> > > >  helper.mk | 4 ++--
> > > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/helper.mk b/helper.mk
> > > > index 79a1da0..2b75182 100644
> > > > --- a/helper.mk
> > > > +++ b/helper.mk
> > > > @@ -35,7 +35,7 @@
> > > > 
> > > >  #    line. Next, append "@" at the end and print the line. Finally,
> > > >  #    append "~" at the end of line. This will make sense in
> > > >  conjunction #    with 6) and 7).
> > > > 
> > > > -# 7) Sort the lines. It is imperative to use LC_COLLATE=C here
> > > > because
> > > > +# 7) Sort the lines. It is imperative to use LC_ALL=C here because
> > > > 
> > > >  #    with this, the "\a" symbol is first and "~" symbol is last. Any
> > > >  #    other symbols fall inbetween. Symbols like "@", which marks the
> > > >  #    end of current line (representing current section) and ".",
> > > >  which
> > > > 
> > > > @@ -57,7 +57,7 @@ $(1): $(2)
> > > > 
> > > >  		-e 's/\.[^\.]\+$$$$//' \
> > > >  		-e ':s /^.\+$$$$/ { p;s/^\(.*\)\.[^\.]*$$$$/\1/;b s }' | \
> > > >  	
> > > >  	sed -n -e 'h;s/$$$$/\a/p;g;s/$$$$/@/p;g;s/$$$$/~/p;' | \
> > > > 
> > > > -	LC_COLLATE=C sort -u | \
> > > > +	LC_ALL=C sort -u | \
> > > 
> > > How will this work? LC_COLLATE is supposed to adjust the behavior of
> > > "sort" command, without it, "sort" will not behave as expected.
> > 
> > Clearly, LC_COLLATE is not enough for it to behave correctly, as seen on
> > my
> > setup.
> > This works because LC_ALL sets LC_COLLATE, as well as all other LC_*
> > environment variables.
> 
> Do you have any reference where I can verify this claim please?

Actually, the problem seems to be a little different - when i have LC_ALL set, 
changing LC_COLLATE doesnt do anything (try running "LC_COLLATE=crazy locale" 
with LC_ALL set)
anyways, the fix remains the same.

Pavel Herrmann

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Marek Vasut]

Dear Pavel Herrmann,

[...]

> > > > > -	LC_COLLATE=C sort -u | \
> > > > > +	LC_ALL=C sort -u | \
> > > > 
> > > > How will this work? LC_COLLATE is supposed to adjust the behavior of
> > > > "sort" command, without it, "sort" will not behave as expected.
> > > 
> > > Clearly, LC_COLLATE is not enough for it to behave correctly, as seen
> > > on my
> > > setup.
> > > This works because LC_ALL sets LC_COLLATE, as well as all other LC_*
> > > environment variables.
> > 
> > Do you have any reference where I can verify this claim please?
> 
> Actually, the problem seems to be a little different - when i have LC_ALL
> set, changing LC_COLLATE doesnt do anything (try running "LC_COLLATE=crazy
> locale" with LC_ALL set)
> anyways, the fix remains the same.

Ugh, can you please provide any relevant documentation? Besides, is the commit 
message wrong now?

Best regards,
Marek Vasut

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Wolfgang Denk]

Dear Pavel Herrmann,

In message <1354470210-30801-1-git-send-email-morpheus.ibis@gmail.com> you wrote:
> Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result is a build
> that has no available commands. Setting LC_ALL=C for the generator script helps.

Setting LC_ALL is a pretty supid thing and can usually be considered
a bug.  Using it as suggested here brings us fromt he frying pan into
the fire.

IMHO it would make more sense to just make sure that LC_ALL is not set
at all, i. e. to explicitly unset it.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
       There is enough for the need of everyone in this world,
       but not for the greed of everyone.     - Mahatma Gandhi

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Marek Vasut]

Dear Wolfgang Denk,

> Dear Pavel Herrmann,
> 
> In message <1354470210-30801-1-git-send-email-morpheus.ibis@gmail.com> you 
wrote:
> > Setting LC_COLLATE=C is not enough if LC_ALL=en_US.utf8. The result is a
> > build that has no available commands. Setting LC_ALL=C for the generator
> > script helps.
> 
> Setting LC_ALL is a pretty supid thing and can usually be considered
> a bug.  Using it as suggested here brings us fromt he frying pan into
> the fire.
> 
> IMHO it would make more sense to just make sure that LC_ALL is not set
> at all, i. e. to explicitly unset it.

Hrm ... shall we introduce a patch that sanitizes the env then? Maybe we can fix 
the LDFLAGS/yocto issue that way too?

Best regards,
Marek Vasut

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Wolfgang Denk]

Dear Marek Vasut,

In message <201212030457.11755.marex@denx.de> you wrote:
> 
> > IMHO it would make more sense to just make sure that LC_ALL is not set
> > at all, i. e. to explicitly unset it.
> 
> Hrm ... shall we introduce a patch that sanitizes the env then? Maybe we can fix 
> the LDFLAGS/yocto issue that way too?

Except for LC_ALL, what else do you have in mind?


I would leave out the LDFLAGS thing here; it is unrelated, and it
bites also for all older versions of U-Boot, and we cannot fix the
past.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
8 Catfish                                               = 1 Octo-puss

[U-Boot] [PATCH] fix linker generated lists on non-C locales

[Marek Vasut]

Dear Wolfgang Denk,

> Dear Marek Vasut,
> 
> In message <201212030457.11755.marex@denx.de> you wrote:
> > > IMHO it would make more sense to just make sure that LC_ALL is not set
> > > at all, i. e. to explicitly unset it.
> > 
> > Hrm ... shall we introduce a patch that sanitizes the env then? Maybe we
> > can fix the LDFLAGS/yocto issue that way too?
> 
> Except for LC_ALL, what else do you have in mind?

If that's the only issue, then only LC_ALL.

> I would leave out the LDFLAGS thing here; it is unrelated, and it
> bites also for all older versions of U-Boot, and we cannot fix the
> past.

That's right.

> Best regards,
> 
> Wolfgang Denk

Best regards,
Marek Vasut