Accept-language test fails on Mac OS

Accept-language test fails on Mac OS

[Michael Blume]

Test #25 'git client sends Accept-Language based on LANGUAGE, LC_ALL,
LC_MESSAGES and LANG' in t5550 fails consistently on my mac, and has
since the test was introduced. Test 26 and 27 ('git client sends
Accept-Language with many preferred languages' and 'git client does
not send Accept-Language') seem fine.

I'm building git with NO_GETTEXT=1, which may be an issue? But in that
case the test should probably be gated on gettext?

Re: Accept-language test fails on Mac OS

[Junio C Hamano]

Michael Blume <blume.mike@gmail.com> writes:

> Test #25 'git client sends Accept-Language based on LANGUAGE, LC_ALL,
> LC_MESSAGES and LANG' in t5550 fails consistently on my mac, and has
> since the test was introduced. Test 26 and 27 ('git client sends
> Accept-Language with many preferred languages' and 'git client does
> not send Accept-Language') seem fine.
>
> I'm building git with NO_GETTEXT=1, which may be an issue? But in that
> case the test should probably be gated on gettext?

I recall queuing a SQUASH??? on top of the posted patch; does these
tests pass with it reverted?

Re: Accept-language test fails on Mac OS

[Michael Blume]

On Fri, Dec 5, 2014 at 2:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Michael Blume <blume.mike@gmail.com> writes:
>
>> Test #25 'git client sends Accept-Language based on LANGUAGE, LC_ALL,
>> LC_MESSAGES and LANG' in t5550 fails consistently on my mac, and has
>> since the test was introduced. Test 26 and 27 ('git client sends
>> Accept-Language with many preferred languages' and 'git client does
>> not send Accept-Language') seem fine.
>>
>> I'm building git with NO_GETTEXT=1, which may be an issue? But in that
>> case the test should probably be gated on gettext?
>
> I recall queuing a SQUASH??? on top of the posted patch; does these
> tests pass with it reverted?

The test fails both on pu and on 7567fad which is prior to the
SQUASH??? commit, so the squash does not seem to change anything.

Re: Accept-language test fails on Mac OS

[Torsten Bögershausen]

On 2014-12-06 00.01, Michael Blume wrote:
> On Fri, Dec 5, 2014 at 2:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Michael Blume <blume.mike@gmail.com> writes:
>>
>>> Test #25 'git client sends Accept-Language based on LANGUAGE, LC_ALL,
>>> LC_MESSAGES and LANG' in t5550 fails consistently on my mac, and has
>>> since the test was introduced. Test 26 and 27 ('git client sends
>>> Accept-Language with many preferred languages' and 'git client does
>>> not send Accept-Language') seem fine.
>>>
>>> I'm building git with NO_GETTEXT=1, which may be an issue? But in that
>>> case the test should probably be gated on gettext?
>>
>> I recall queuing a SQUASH??? on top of the posted patch; does these
>> tests pass with it reverted?
> 
> The test fails both on pu and on 7567fad which is prior to the
> SQUASH??? commit, so the squash does not seem to change anything.
pu@c83418638e0c326f:
t5550 fails here under Debian Linux,
I haven't digged further

not ok 25 - git client sends Accept-Language based on LANGUAGE, LC_ALL, LC_MESSAGES and LANG
#       
#               check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
#               check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
#               check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
#               check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8
#       

Re: Accept-language test fails on Mac OS

[Yi, EungJun]

Could you tell me your locale information from executing 'locale'
command and the verbose message you can get by accessing any git
repository via HTTP protocol? (e.g. GIT_CURL_VERBOSE=1 git clone
http://github.com/foo/bar )

I think the failures are related with your locale information.

On Sat, Dec 6, 2014 at 6:50 PM, Torsten Bögershausen <tboegi@web.de> wrote:
> On 2014-12-06 00.01, Michael Blume wrote:
>> On Fri, Dec 5, 2014 at 2:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>> Michael Blume <blume.mike@gmail.com> writes:
>>>
>>>> Test #25 'git client sends Accept-Language based on LANGUAGE, LC_ALL,
>>>> LC_MESSAGES and LANG' in t5550 fails consistently on my mac, and has
>>>> since the test was introduced. Test 26 and 27 ('git client sends
>>>> Accept-Language with many preferred languages' and 'git client does
>>>> not send Accept-Language') seem fine.
>>>>
>>>> I'm building git with NO_GETTEXT=1, which may be an issue? But in that
>>>> case the test should probably be gated on gettext?
>>>
>>> I recall queuing a SQUASH??? on top of the posted patch; does these
>>> tests pass with it reverted?
>>
>> The test fails both on pu and on 7567fad which is prior to the
>> SQUASH??? commit, so the squash does not seem to change anything.
> pu@c83418638e0c326f:
> t5550 fails here under Debian Linux,
> I haven't digged further
>
> not ok 25 - git client sends Accept-Language based on LANGUAGE, LC_ALL, LC_MESSAGES and LANG
> #
> #               check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
> #               check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
> #               check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
> #               check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8
> #
>

Re: Accept-language test fails on Mac OS

[Torsten Bögershausen]

On 2014-12-06 20.44, Yi, EungJun wrote:
> Could you tell me your locale information from executing 'locale'
> command and the verbose message you can get by accessing any git
> repository via HTTP protocol? (e.g. GIT_CURL_VERBOSE=1 git clone
> http://github.com/foo/bar )
locale -a
C
C.UTF-8
en_US.utf8
icelandic
is_IS
is_IS.iso88591
is_IS.utf8
POSIX

-----------------------------
After patching like this:
diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh
index fc6e7fc..1e42b42 100755
--- a/t/t5550-http-fetch-dumb.sh
+++ b/t/t5550-http-fetch-dumb.sh
@@ -205,8 +205,8 @@ check_language () {
        LANG=$5 \
        git ls-remote "$HTTPD_URL/dumb/repo.git" 2>&1 |
        tr -d '\015' |
-       sort -u >stderr &&
-       grep -i ^Accept-Language: stderr >actual &&
+       sort -u >stderr
+       grep -i ^Accept-Language: stderr >actual
        test_cmp expect actual
 }
-----------
I get this:


expecting success: 
        check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
        check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
        check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
        check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8

--- expect      2014-12-06 21:00:59.000000000 +0000
+++ actual      2014-12-06 21:00:59.000000000 +0000
@@ -1 +0,0 @@
-Accept-Language: de-DE, *;q=0.1
not ok 25 - git client sends Accept-Language based on LANGUAGE, LC_ALL, LC_MESSAGES and LANG
#       
#               check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
#               check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
#               check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
#               check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8
#       

Re: Accept-language test fails on Mac OS

[Yi, EungJun]

Thank you for providing useful information to fix the failures.

On Sun, Dec 7, 2014 at 6:04 AM, Torsten Bögershausen <tboegi@web.de> wrote:
> On 2014-12-06 20.44, Yi, EungJun wrote:
>> Could you tell me your locale information from executing 'locale'
>> command and the verbose message you can get by accessing any git
>> repository via HTTP protocol? (e.g. GIT_CURL_VERBOSE=1 git clone
>> http://github.com/foo/bar )
> locale -a
> C
> C.UTF-8
> en_US.utf8
> icelandic
> is_IS
> is_IS.iso88591
> is_IS.utf8
> POSIX

I'm sorry for bothering you, but could you tell me the result of
"locale" command without "-a" option? What I want to know is locale
environment variables and its values, so I want to reproduce the test
failures on my laptop.

>
> -----------------------------
> After patching like this:
> diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh
> index fc6e7fc..1e42b42 100755
> --- a/t/t5550-http-fetch-dumb.sh
> +++ b/t/t5550-http-fetch-dumb.sh
> @@ -205,8 +205,8 @@ check_language () {
>         LANG=$5 \
>         git ls-remote "$HTTPD_URL/dumb/repo.git" 2>&1 |
>         tr -d '\015' |
> -       sort -u >stderr &&
> -       grep -i ^Accept-Language: stderr >actual &&
> +       sort -u >stderr
> +       grep -i ^Accept-Language: stderr >actual
>         test_cmp expect actual
>  }
> -----------
> I get this:
>
>
> expecting success:
>         check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
>         check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
>         check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
>         check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8
>
> --- expect      2014-12-06 21:00:59.000000000 +0000
> +++ actual      2014-12-06 21:00:59.000000000 +0000
> @@ -1 +0,0 @@
> -Accept-Language: de-DE, *;q=0.1
> not ok 25 - git client sends Accept-Language based on LANGUAGE, LC_ALL, LC_MESSAGES and LANG
> #
> #               check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
> #               check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
> #               check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
> #               check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8
> #
>
>

The test seems to fail if LANGUAGE="". I will dig it. Thanks.

Re: Accept-language test fails on Mac OS

[Torsten Bögershausen]

On 07.12.14 07:54, Yi, EungJun wrote:
> I'm sorry for bothering you, but could you tell me the result of
> "locale" command without "-a" option? What I want to know is locale
> environment variables and its values, so I want to reproduce the test
> failures on my laptop.
> 
(Just for completeness:)
 locale
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Re: Accept-language test fails on Mac OS

[Jeff King]

On Sat, Dec 06, 2014 at 10:04:06PM +0100, Torsten Bögershausen wrote:

> I get this:
> 
> 
> expecting success: 
>         check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
>         check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
>         check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
>         check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8
> 
> --- expect      2014-12-06 21:00:59.000000000 +0000
> +++ actual      2014-12-06 21:00:59.000000000 +0000
> @@ -1 +0,0 @@
> -Accept-Language: de-DE, *;q=0.1
> not ok 25 - git client sends Accept-Language based on LANGUAGE, LC_ALL, LC_MESSAGES and LANG

I can reproduce the same problem here (Debian unstable). I actually ran
into three issues (aside from needing to use Junio's SQUASH commit, to
avoid the "\r" bash-ism):

  1. I couldn't build without including locale.h, for the
     definition of setlocale() and the LC_MESSAGES constant (both used
     in get_preferred_languages).

     I'm not sure what portability issues there are with including it
     unconditionally. Should this possibly be tied into gettext.c, which
     already uses setlocale?

  2. The call to setlocale(LC_MESSAGES, NULL) in get_preferred_languages
     always returns "C" for me. This seems related to building with
     NO_GETTEXT (which I typically do), as we never init setlocale
     if NO_GETTEXT is set. This program demonstrates it:

	#include <stdio.h>
	#include <string.h>
	#include <locale.h>
	
	int main(int argc, char **argv)
	{
		if (argv[1] && !strcmp(argv[1], "init"))
			setlocale(LC_MESSAGES, "");
		printf("%s", setlocale(LC_MESSAGES, NULL));
		return 0;
	}

     If I run it as "LANG=en_US.UTF-8 ./a.out", it prints "C". If I run
     it as "LANG=en_US.UTF-8 ./a.out init", it prints "en_US.UTF-8". I
     think we either need to start unconditionally calling setlocale()
     as we do in git_setup_gettext, or we need to tie your feature to
     using gettext.

     This is what causes the failure of the de-DE test for me; building
     without NO_GETTEXT makes it work. Note that this doesn't affect the
     first test for ko-KR, because that test sets LANGUAGE, which we
     read ourselves (so we never make a setlocale() call).

  3. Even building with NO_GETTEXT, setlocale() does not want to
     report ja_JP.UTF-8 for me, making the third test fail.

     I think the issue is that I do not build the ja_JP locale on my
     system. Running "dpkg-reconfigure locales" and asking it to build
     ja_JP.UTF-8 makes the test pass. This is somewhat of a Debian-ism.
     From "man locale-gen":

       By default, the locale package which provides the base support
       for localisation of libc-based programs does not contain usable
       localisation files for every supported language. This limitation
       has became necessary because of the substantial size of such
       files and the large number of languages supported by libc. As a
       result, Debian uses a special mechanism where we prepare the
       actual localisation files on the target host and distribute only
       the templates for them.

     I suspect it is inherited by Debian derivatives like Ubuntu. But I
     also don't know that we can count on other platforms having all of
     the locales either (e.g., they may ship them as separate packages,
     not all of which are installed).

     So I'm not sure of an easy way around this. You want 4 separate
     locales to thoroughly test, but you cannot rely on any particular
     locale being present on the user's system.

     Note that this is just a problem with the tests, probably not with
     the feature itself. Presumably people setting LANG=ja_JP actually
     have that locale on their system (though technically this feature
     is about asking the _server_ to use that language, it seems like
     you would do so because you were using that language locally, too).

-Peff

Re: Accept-language test fails on Mac OS

[Torsten Bögershausen]

On 07.12.14 08:18, Jeff King wrote:
> On Sat, Dec 06, 2014 at 10:04:06PM +0100, Torsten Bögershausen wrote:
> 
>> I get this:
>>
>>
>> expecting success: 
>>         check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
>>         check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
>>         check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
>>         check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8
>>
>> --- expect      2014-12-06 21:00:59.000000000 +0000
>> +++ actual      2014-12-06 21:00:59.000000000 +0000
>> @@ -1 +0,0 @@
>> -Accept-Language: de-DE, *;q=0.1
>> not ok 25 - git client sends Accept-Language based on LANGUAGE, LC_ALL, LC_MESSAGES and LANG
> 
> I can reproduce the same problem here (Debian unstable). I actually ran
> into three issues (aside from needing to use Junio's SQUASH commit, to
> avoid the "\r" bash-ism):
> 
>   1. I couldn't build without including locale.h, for the
>      definition of setlocale() and the LC_MESSAGES constant (both used
>      in get_preferred_languages).
> 
>      I'm not sure what portability issues there are with including it
>      unconditionally. Should this possibly be tied into gettext.c, which
>      already uses setlocale?
> 
>   2. The call to setlocale(LC_MESSAGES, NULL) in get_preferred_languages
>      always returns "C" for me. This seems related to building with
>      NO_GETTEXT (which I typically do), as we never init setlocale
>      if NO_GETTEXT is set. This program demonstrates it:
> 
> 	#include <stdio.h>
> 	#include <string.h>
> 	#include <locale.h>
> 	
> 	int main(int argc, char **argv)
> 	{
> 		if (argv[1] && !strcmp(argv[1], "init"))
> 			setlocale(LC_MESSAGES, "");
> 		printf("%s", setlocale(LC_MESSAGES, NULL));
> 		return 0;
> 	}
> 
>      If I run it as "LANG=en_US.UTF-8 ./a.out", it prints "C". If I run
>      it as "LANG=en_US.UTF-8 ./a.out init", it prints "en_US.UTF-8". I
>      think we either need to start unconditionally calling setlocale()
>      as we do in git_setup_gettext, or we need to tie your feature to
>      using gettext.
> 
>      This is what causes the failure of the de-DE test for me; building
>      without NO_GETTEXT makes it work. Note that this doesn't affect the
>      first test for ko-KR, because that test sets LANGUAGE, which we
>      read ourselves (so we never make a setlocale() call).
> 
>   3. Even building with NO_GETTEXT, setlocale() does not want to
>      report ja_JP.UTF-8 for me, making the third test fail.
> 
>      I think the issue is that I do not build the ja_JP locale on my
>      system. Running "dpkg-reconfigure locales" and asking it to build
>      ja_JP.UTF-8 makes the test pass. This is somewhat of a Debian-ism.
>      From "man locale-gen":
> 
>        By default, the locale package which provides the base support
>        for localisation of libc-based programs does not contain usable
>        localisation files for every supported language. This limitation
>        has became necessary because of the substantial size of such
>        files and the large number of languages supported by libc. As a
>        result, Debian uses a special mechanism where we prepare the
>        actual localisation files on the target host and distribute only
>        the templates for them.
> 
>      I suspect it is inherited by Debian derivatives like Ubuntu. But I
>      also don't know that we can count on other platforms having all of
>      the locales either (e.g., they may ship them as separate packages,
>      not all of which are installed).
> 
>      So I'm not sure of an easy way around this. You want 4 separate
>      locales to thoroughly test, but you cannot rely on any particular
>      locale being present on the user's system.
> 
>      Note that this is just a problem with the tests, probably not with
>      the feature itself. Presumably people setting LANG=ja_JP actually
>      have that locale on their system (though technically this feature
>      is about asking the _server_ to use that language, it seems like
>      you would do so because you were using that language locally, too).
> 
> -Peff
> --
(I remember debugging t0204 some time ago)

We may get inspiration from this, either how to adjust the locale to be used,
or if the test should be skipped.

 grep -l is_IS *
lib-gettext.sh
t0200-gettext-basic.sh
t0203-gettext-setlocale-sanity.sh
t0204-gettext-reencode-sanity.sh

Re: Accept-language test fails on Mac OS

[Christian Couder]

On Sun, Dec 7, 2014 at 8:18 AM, Jeff King <peff@peff.net> wrote:
> On Sat, Dec 06, 2014 at 10:04:06PM +0100, Torsten Bögershausen wrote:
>
>> I get this:
>>
>>
>> expecting success:
>>         check_language "ko-KR, *;q=0.1" ko_KR.UTF-8 de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
>>         check_language "de-DE, *;q=0.1" ""          de_DE.UTF-8 ja_JP.UTF-8 en_US.UTF-8 &&
>>         check_language "ja-JP, *;q=0.1" ""          ""          ja_JP.UTF-8 en_US.UTF-8 &&
>>         check_language "en-US, *;q=0.1" ""          ""          ""          en_US.UTF-8
>>
>> --- expect      2014-12-06 21:00:59.000000000 +0000
>> +++ actual      2014-12-06 21:00:59.000000000 +0000
>> @@ -1 +0,0 @@
>> -Accept-Language: de-DE, *;q=0.1
>> not ok 25 - git client sends Accept-Language based on LANGUAGE, LC_ALL, LC_MESSAGES and LANG
>
> I can reproduce the same problem here (Debian unstable). I actually ran
> into three issues (aside from needing to use Junio's SQUASH commit, to
> avoid the "\r" bash-ism):
>
>   1. I couldn't build without including locale.h, for the
>      definition of setlocale() and the LC_MESSAGES constant (both used
>      in get_preferred_languages).
>
>      I'm not sure what portability issues there are with including it
>      unconditionally. Should this possibly be tied into gettext.c, which
>      already uses setlocale?

Yeah, pu build is broken on Ubuntu 14.04 too, because of
7567fad2431eb38291fd74a70f603e5746c6f728 (http: send Accept-Language
header if possible).

Thanks,
Christian.