Allow dash to build with i18n version of Coreutils

Allow dash to build with i18n version of Coreutils

[Matthew Burgess]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

Hi,

My system has Coreutils-7.4 compiled with the i18n patch from
http://cvs.fedoraproject.org/viewvc/devel/coreutils/coreutils-i18n.patch.

Using this to compile dash, when in an en_GB.UTF-8 locale, I get the following error:

if gcc -DHAVE_CONFIG_H -I. -I. -I..  -include ../config.h -DBSD=1 -DSHELL -DIFS_BROKEN  -Wall -g -O2 -MT eval.o -MD -MP -MF ".deps/eval.Tpo" -c -o eval.o eval.c; \
        then mv -f ".deps/eval.Tpo" ".deps/eval.Po"; else rm -f ".deps/eval.Tpo"; exit 1; fi
eval.c: In function ‘evalcommand’:
eval.c:810: error: ‘EXECCMD’ undeclared (first use in this function)
eval.c:810: error: (Each undeclared identifier is reported only once
eval.c:810: error: for each function it appears in.)
eval.c:812: error: ‘COMMANDCMD’ undeclared (first use in this function)
make[3]: *** [eval.o] Error 1

This is because the src/mkbuiltins script ends up generating an incomplete
src/builtins.h file.  This, in turn, is caused by 'sort -u -k 3,3' not
working correctly.

The attached patch fixes/works around things by setting LC_CTYPE=C, thus
overriding the build environment (in a similar manner to the earlier call to
'sort' in that same script).

Regards,

Matt.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dash-0.5.5.1-sort_utf8-1.patch --]
[-- Type: text/x-diff; name="dash-0.5.5.1-sort_utf8-1.patch", Size: 742 bytes --]

Submitted by: Matt Burgess (matthew_at_linuxfromscratch_dot_org)
Date: 2009-08-15
Initial Package Version: 0.5.5.1
Upstream Status: Submitted
Origin: Matt Burgess
Description: In UTF-8 locale's with an 

diff -Naur dash-0.5.5.1.orig/src/mkbuiltins dash-0.5.5.1/src/mkbuiltins
--- dash-0.5.5.1.orig/src/mkbuiltins	2009-01-13 23:37:13.000000000 +0000
+++ dash-0.5.5.1/src/mkbuiltins	2009-08-15 17:31:19.000000000 +0000
@@ -83,7 +83,7 @@
  */
 
 !
-sed 's/	-[a-z]*//' $temp2 | nl -v 0 | sort -u -k 3,3 |
+sed 's/	-[a-z]*//' $temp2 | nl -v 0 | LC_CTYPE=C sort -u -k 3,3 |
 tr abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ |
 	awk '{	printf "#define %s (builtincmd + %d)\n", $3, $1}'
 printf '\n#define NUMBUILTINS %d\n' $(wc -l < $temp2)

Re: Allow dash to build with i18n version of Coreutils

[Herbert Xu]

On Sat, Aug 15, 2009 at 06:07:16PM +0000, Matthew Burgess wrote:
> 
> My system has Coreutils-7.4 compiled with the i18n patch from
> http://cvs.fedoraproject.org/viewvc/devel/coreutils/coreutils-i18n.patch.
> 
> Using this to compile dash, when in an en_GB.UTF-8 locale, I get the following error:
> 
> if gcc -DHAVE_CONFIG_H -I. -I. -I..  -include ../config.h -DBSD=1 -DSHELL -DIFS_BROKEN  -Wall -g -O2 -MT eval.o -MD -MP -MF ".deps/eval.Tpo" -c -o eval.o eval.c; \
>         then mv -f ".deps/eval.Tpo" ".deps/eval.Po"; else rm -f ".deps/eval.Tpo"; exit 1; fi
> eval.c: In function ‘evalcommand’:
> eval.c:810: error: ‘EXECCMD’ undeclared (first use in this function)
> eval.c:810: error: (Each undeclared identifier is reported only once
> eval.c:810: error: for each function it appears in.)
> eval.c:812: error: ‘COMMANDCMD’ undeclared (first use in this function)
> make[3]: *** [eval.o] Error 1
> 
> This is because the src/mkbuiltins script ends up generating an incomplete
> src/builtins.h file.  This, in turn, is caused by 'sort -u -k 3,3' not
> working correctly.
> 
> The attached patch fixes/works around things by setting LC_CTYPE=C, thus
> overriding the build environment (in a similar manner to the earlier call to
> 'sort' in that same script).

Thank you.  I've applied the patch but replaced LC_CTYPE with
LC_COLLATE.
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Allow dash to build with i18n version of Coreutils

[Matthew Burgess]

On Mon, 31 Aug 2009 20:27:28 +1000, Herbert Xu <herbert@gondor.apana.org.au> wrote:

> Thank you.  I've applied the patch but replaced LC_CTYPE with
> LC_COLLATE.

Hi Herbert, for some reason, 'sort' doesn't appear to be honouring LC_COLLATE; only
LC_ALL & LC_CTYPE appears to get things working with this particular invocation of
'sort':

su-4.0$ echo $LANG 
en_GB.UTF-8

su-4.0$ sort -u -k 3,3 builtins
     0  .       -s      dotcmd

su-4.0$ LC_ALL=C sort -u -k 3,3 builtins
    21  local   -a      localcmd
    14  export  -as     exportcmd
     3  alias   -au     aliascmd
     0  .       -s      dotcmd
     4  bg      -u      bgcmd
     7  chdir   cdcmd
    10  echo    echocmd
    18  hash    hashcmd
    22  printf  printfcmd
    23  pwd     pwdcmd
     2  [       testcmd
    33  type    typecmd
    34  ulimit  ulimitcmd

su-4.0$ LC_CTYPE=C sort -u -k 3,3 builtins
    21  local   -a      localcmd
    14  export  -as     exportcmd
     3  alias   -au     aliascmd
     7  chdir   cdcmd
    10  echo    echocmd
    18  hash    hashcmd
    22  printf  printfcmd
    23  pwd     pwdcmd
     0  .       -s      dotcmd
     2  [       testcmd
    33  type    typecmd
     4  bg      -u      bgcmd
    34  ulimit  ulimitcmd

su-4.0$ LC_COLLATE=C sort -u -k 3,3 builtins
     0  .       -s      dotcmd

Regards,

Matt.

Re: Allow dash to build with i18n version of Coreutils

[Herbert Xu]

On Mon, Aug 31, 2009 at 08:19:24AM -0600, Matthew Burgess wrote:
> 
> su-4.0$ echo $LANG 
> en_GB.UTF-8
> 
> su-4.0$ sort -u -k 3,3 builtins
>      0  .       -s      dotcmd
> 
> su-4.0$ LC_ALL=C sort -u -k 3,3 builtins
>     21  local   -a      localcmd
>     14  export  -as     exportcmd
>      3  alias   -au     aliascmd
>      0  .       -s      dotcmd
>      4  bg      -u      bgcmd
>      7  chdir   cdcmd
>     10  echo    echocmd
>     18  hash    hashcmd
>     22  printf  printfcmd
>     23  pwd     pwdcmd
>      2  [       testcmd
>     33  type    typecmd
>     34  ulimit  ulimitcmd
> 
> su-4.0$ LC_CTYPE=C sort -u -k 3,3 builtins
>     21  local   -a      localcmd
>     14  export  -as     exportcmd
>      3  alias   -au     aliascmd
>      7  chdir   cdcmd
>     10  echo    echocmd
>     18  hash    hashcmd
>     22  printf  printfcmd
>     23  pwd     pwdcmd
>      0  .       -s      dotcmd
>      2  [       testcmd
>     33  type    typecmd
>      4  bg      -u      bgcmd
>     34  ulimit  ulimitcmd
> 
> su-4.0$ LC_COLLATE=C sort -u -k 3,3 builtins
>      0  .       -s      dotcmd

Weird, it works here:

$ LANG=en_GB.UTF-8 LC_COLLATE=C sort -u -k 3,3 /tmp/foo
     3  alias   aliascmd
     4  bg      bgcmd
     5  break   breakcmd
     6  cd      cdcmd
     8  command commandcmd
     0  .       dotcmd
    10  echo    echocmd
    11  eval    evalcmd
    12  exec    execcmd
    13  exit    exitcmd
    14  export  exportcmd
    15  false   falsecmd
    16  fg      fgcmd
    17  getopts getoptscmd
    18  hash    hashcmd
    19  jobs    jobscmd
    20  kill    killcmd
    21  local   localcmd
    22  printf  printfcmd
    23  pwd     pwdcmd
    24  read    readcmd
    26  return  returncmd
    27  set     setcmd
    28  shift   shiftcmd
     2  [       testcmd
    30  times   timescmd
    31  trap    trapcmd
     1  :       truecmd
    33  type    typecmd
    34  ulimit  ulimitcmd
    35  umask   umaskcmd
    36  unalias unaliascmd
    37  unset   unsetcmd
    38  wait    waitcmd
$

What version of glibc/sort are you using?

In fact there is definitely something screwed because usually
LANG=en_* only changes the way upper-case and lower-case letters
get sorted rather than what you saw above.

Oh I see what's going on.  In fact it's sed that's not working for
you, not sort.  Let me find a way around that.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Allow dash to build with i18n version of Coreutils

[Matthew Burgess]

On Tue, 1 Sep 2009 13:15:57 +1000, Herbert Xu <herbert@gondor.apana.org.au> wrote:
> On Mon, Aug 31, 2009 at 08:19:24AM -0600, Matthew Burgess wrote:
> 
> Oh I see what's going on.  In fact it's sed that's not working for
> you, not sort.  Let me find a way around that.

Ah, well spotted.  The following is enough to get the correct output from sed(1):

sed 's/\t-[a-z]*//' $temp2

This is with sed-4.2.1 compiled against Glibc-2.10.1.

Running through 'nl -v 0', I end up with:

     0	.	dotcmd
     1	:	truecmd
     2	[	testcmd
     3	alias	aliascmd
     4	bg	bgcmd
     5	break	breakcmd
     6	cd	cdcmd
     7	chdir	cdcmd
     8	command	commandcmd
     9	continue	breakcmd
    10	echo	echocmd
    11	eval	evalcmd
    12	exec	execcmd
    13	exit	exitcmd
    14	export	exportcmd
    15	false	falsecmd
    16	fg	fgcmd
    17	getopts	getoptscmd
    18	hash	hashcmd
    19	jobs	jobscmd
    20	kill	killcmd
    21	local	localcmd
    22	printf	printfcmd
    23	pwd	pwdcmd
    24	read	readcmd
    25	readonly	exportcmd
    26	return	returncmd
    27	set	setcmd
    28	shift	shiftcmd
    29	test	testcmd
    30	times	timescmd
    31	trap	trapcmd
    32	true	truecmd
    33	type	typecmd
    34	ulimit	ulimitcmd
    35	umask	umaskcmd
    36	unalias	unaliascmd
    37	unset	unsetcmd
    38	wait	waitcmd

That still doesn't help with the sort(1) output though (from coreutils-7.5), which
still doesn't appear to be honouring LC_COLLATE!  I suspect a bug in our i18n patch
for Coreutils, which I'll try to take a look at over the next few days.

Thanks,

Matt.