[Qemu-devel] qapi escape-too-big test doesn't work if LANG=C ?

[Qemu-devel] qapi escape-too-big test doesn't work if LANG=C ?

[Peter Maydell]

I recently tweaked my build scripts to run with LANG=C (trying
to suppress gcc's irritating habit of using smartquotes rather
than plain old ''). This seems to result in an error running
the qapi-schema/escape-too-big test:

PYTHONPATH=/home/petmay01/linaro/qemu-for-merges/scripts python3 -B
/home/petmay01/linaro/qemu-for-merges/tests/qapi-schema/test-qapi.py
/home/petmay01/linaro/qemu-for-merges/tests/qapi-schema/escape-too-big.json
>tests/qapi-schema/escape-too-big.test.out
2>tests/qapi-schema/escape-too-big.test.err; echo $?
>tests/qapi-schema/escape-too-big.test.exit
1c1,10
< tests/qapi-schema/escape-too-big.json:3:14: For now, \u escape only
supports non-zero values up to \u007f
---
> Traceback (most recent call last):
>   File "tests/qapi-schema/test-qapi.py", line 64, in <module>
>     schema = QAPISchema(sys.argv[1])
>   File "scripts/qapi/common.py", line 1492, in __init__
>     parser = QAPISchemaParser(open(fname, 'r'))
>   File "scripts/qapi/common.py", line 264, in __init__
>     self.src = fp.read()
>   File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
>     return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 61: ordinal not in range(128)
/home/petmay01/linaro/qemu-for-merges/tests/Makefile.include:927:
recipe for target 'check-tests/qapi-schema/escape-too-big.json' failed

thanks
-- PMM

Re: [Qemu-devel] qapi escape-too-big test doesn't work if LANG=C ?

[Daniel P. Berrangé]

On Mon, Mar 19, 2018 at 10:37:12AM +0000, Peter Maydell wrote:
> I recently tweaked my build scripts to run with LANG=C (trying
> to suppress gcc's irritating habit of using smartquotes rather
> than plain old ''). This seems to result in an error running
> the qapi-schema/escape-too-big test:
> 
> PYTHONPATH=/home/petmay01/linaro/qemu-for-merges/scripts python3 -B
> /home/petmay01/linaro/qemu-for-merges/tests/qapi-schema/test-qapi.py
> /home/petmay01/linaro/qemu-for-merges/tests/qapi-schema/escape-too-big.json
> >tests/qapi-schema/escape-too-big.test.out
> 2>tests/qapi-schema/escape-too-big.test.err; echo $?
> >tests/qapi-schema/escape-too-big.test.exit
> 1c1,10
> < tests/qapi-schema/escape-too-big.json:3:14: For now, \u escape only
> supports non-zero values up to \u007f
> ---
> > Traceback (most recent call last):
> >   File "tests/qapi-schema/test-qapi.py", line 64, in <module>
> >     schema = QAPISchema(sys.argv[1])
> >   File "scripts/qapi/common.py", line 1492, in __init__
> >     parser = QAPISchemaParser(open(fname, 'r'))
> >   File "scripts/qapi/common.py", line 264, in __init__
> >     self.src = fp.read()
> >   File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
> >     return codecs.ascii_decode(input, self.errors)[0]
> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 61: ordinal not in range(128)
> /home/petmay01/linaro/qemu-for-merges/tests/Makefile.include:927:
> recipe for target 'check-tests/qapi-schema/escape-too-big.json' failed

So your "C" locale will be non-UTF-8, except on OS-X where the "C" locale
is UTF-8 by default.

Unfortunately while POSIX expects the "C" locale to be 8-bit cleanup,
Python by default will reject any characters outside the 7-bit range
with its "ascii" codec. So this is ultimately a python bug, but there's
little we can do about that given how widely deployed the bug is.

To workaround this problem in other applications what I have done is
add the following to Makefiles before invoking python:

    LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 

The LC_ALL= bit is needed because if the user has set LC_ALL themselves
it will override LANG and all other LC_* variables. Setting LANG=C is
not strictly needed, as LC_CTYPE will override it.

CC'ing Eric since he was involved in the discussions about this bug
in other libvirt related apps.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] qapi escape-too-big test doesn't work if LANG=C ?

[Eric Blake]

On 03/19/2018 06:20 AM, Daniel P. Berrangé wrote:
> On Mon, Mar 19, 2018 at 10:37:12AM +0000, Peter Maydell wrote:
>> I recently tweaked my build scripts to run with LANG=C (trying
>> to suppress gcc's irritating habit of using smartquotes rather
>> than plain old ''). This seems to result in an error running
>> the qapi-schema/escape-too-big test:
>>
>> PYTHONPATH=/home/petmay01/linaro/qemu-for-merges/scripts python3 -B
>> /home/petmay01/linaro/qemu-for-merges/tests/qapi-schema/test-qapi.py
>> /home/petmay01/linaro/qemu-for-merges/tests/qapi-schema/escape-too-big.json
>>> tests/qapi-schema/escape-too-big.test.out
>> 2>tests/qapi-schema/escape-too-big.test.err; echo $?
>>> tests/qapi-schema/escape-too-big.test.exit
>> 1c1,10
>> < tests/qapi-schema/escape-too-big.json:3:14: For now, \u escape only
>> supports non-zero values up to \u007f
>> ---
>>> Traceback (most recent call last):
>>>    File "tests/qapi-schema/test-qapi.py", line 64, in <module>
>>>      schema = QAPISchema(sys.argv[1])
>>>    File "scripts/qapi/common.py", line 1492, in __init__
>>>      parser = QAPISchemaParser(open(fname, 'r'))
>>>    File "scripts/qapi/common.py", line 264, in __init__
>>>      self.src = fp.read()
>>>    File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
>>>      return codecs.ascii_decode(input, self.errors)[0]
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 61: ordinal not in range(128)
>> /home/petmay01/linaro/qemu-for-merges/tests/Makefile.include:927:
>> recipe for target 'check-tests/qapi-schema/escape-too-big.json' failed
> 
> So your "C" locale will be non-UTF-8, except on OS-X where the "C" locale
> is UTF-8 by default.
> 
> Unfortunately while POSIX expects the "C" locale to be 8-bit cleanup,
> Python by default will reject any characters outside the 7-bit range
> with its "ascii" codec. So this is ultimately a python bug, but there's
> little we can do about that given how widely deployed the bug is.

Yep, the bug is in python.  And we've already worked around it elsewhere 
in qemu; see commit d4e5ec877.

> 
> To workaround this problem in other applications what I have done is
> add the following to Makefiles before invoking python:
> 
>      LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8
> 
> The LC_ALL= bit is needed because if the user has set LC_ALL themselves
> it will override LANG and all other LC_* variables. Setting LANG=C is
> not strictly needed, as LC_CTYPE will override it.
> 
> CC'ing Eric since he was involved in the discussions about this bug
> in other libvirt related apps.

So the question is how to fix the test to not trip the python bugs.  And 
the only non-7-bit-clean byte in that file is in a comment, so maybe the 
simplest stupid thing that would work for now is to rewrite the comment 
to drop the é and instead use pure ASCII.  A nicer fix would be teaching 
scripts/qapi/common.py to read input in an 8-bit-clean manner regardless 
of locale, but that's more invasive (and I'd rather have Markus' help on 
that front), so for 2.12, I'll just do the obvious bug fix of avoiding 
the problem by skipping the unfortunate comment.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org