[Qemu-devel] [PATCH for-2.12] qapi: Force UTF8 encoding when parsing qapi files

[Qemu-devel] [PATCH for-2.12] qapi: Force UTF8 encoding when parsing qapi files

[Eric Blake]

Commit d4e5ec877 already fixed things to work around Python 3's
lame bug of having LC_ALL=C not be 8-bit clean, when parsing the
main QMP qapi files; but failed to do likewise in the tests
directory.  As a result, running 'LC_ALL=C make check' fails on
escape-too-big and unicode-str when using python 3 with a nasty
stack trace instead of the intended graceful error message that
QAPI doesn't yet support 8-bit data (the two tests contain
Unicode é, when parsed in UTF-8; they represent something
different when parsed in a proper single-byte C locale, but that
doesn't matter to the error message printed out, provided that
brain-dead Python hasn't first choked on the input instead of
being 8-bit clean).

Ideally, we'd teach the qapi generator scripts to automatically
slurp things in using UTF-8 regardless of locale, and to honor
content that is not limited to 7 bit data rather than gracefully
erroring out; but until then, since our graceful error depends
on python parsing 8-bit data (even if nothing we generate uses
8-bit data), our quick fix is to use the right locale when
running these tests.

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Eric Blake <eblake@redhat.com>
---

Will go through my QAPI tree for post-freeze bug fixes

 tests/Makefile.include | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 412aed1737f..ea1f1bf0965 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -668,13 +668,13 @@ tests/test-qapi-events.c tests/test-qapi-events.h \
 tests/test-qapi-introspect.c tests/test-qapi-introspect.h: \
 tests/test-qapi-gen-timestamp ;
 tests/test-qapi-gen-timestamp: $(SRC_PATH)/tests/qapi-schema/qapi-schema-test.json $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+	$(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
 		-o tests -p "test-" $<, \
 		"GEN","$(@:%-timestamp=%)")
 	@>$@

 tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-good.json $(qapi-py)
-	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+	$(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
 		-o tests/qapi-schema -p "doc-good-" $<, \
 		"GEN","$@")
 	@mv tests/qapi-schema/doc-good-qapi-doc.texi $@
@@ -925,7 +925,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		$(PYTHON_UTF8) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
 		$^ >$*.test.out 2>$*.test.err; \
 		echo $$? >$*.test.exit, \
 		"TEST","$*.out")
-- 
2.14.3

Re: [Qemu-devel] [PATCH for-2.12] qapi: Force UTF8 encoding when parsing qapi files

[Daniel P. Berrangé]

On Mon, Mar 19, 2018 at 03:50:40PM -0500, Eric Blake wrote:
> Commit d4e5ec877 already fixed things to work around Python 3's
> lame bug of having LC_ALL=C not be 8-bit clean, when parsing the
> main QMP qapi files; but failed to do likewise in the tests
> directory.  As a result, running 'LC_ALL=C make check' fails on
> escape-too-big and unicode-str when using python 3 with a nasty
> stack trace instead of the intended graceful error message that
> QAPI doesn't yet support 8-bit data (the two tests contain
> Unicode é, when parsed in UTF-8; they represent something
> different when parsed in a proper single-byte C locale, but that
> doesn't matter to the error message printed out, provided that
> brain-dead Python hasn't first choked on the input instead of
> being 8-bit clean).
> 
> Ideally, we'd teach the qapi generator scripts to automatically
> slurp things in using UTF-8 regardless of locale, and to honor
> content that is not limited to 7 bit data rather than gracefully
> erroring out; but until then, since our graceful error depends
> on python parsing 8-bit data (even if nothing we generate uses
> 8-bit data), our quick fix is to use the right locale when
> running these tests.
> 
> Reported-by: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Eric Blake <eblake@redhat.com>
> ---
> 
> Will go through my QAPI tree for post-freeze bug fixes
> 
>  tests/Makefile.include | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|