From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:52763)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <lersek@redhat.com>) id 1VBXUz-0004d6-0z
	for qemu-devel@nongnu.org; Mon, 19 Aug 2013 18:00:35 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <lersek@redhat.com>) id 1VBXUt-0006xV-KU
	for qemu-devel@nongnu.org; Mon, 19 Aug 2013 18:00:28 -0400
Received: from mx1.redhat.com ([209.132.183.28]:49467)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <lersek@redhat.com>) id 1VBXUt-0006xI-D3
	for qemu-devel@nongnu.org; Mon, 19 Aug 2013 18:00:23 -0400
Message-ID: <521295F2.90102@redhat.com>
Date: Tue, 20 Aug 2013 00:02:26 +0200
From: Laszlo Ersek <lersek@redhat.com>
MIME-Version: 1.0
References: <1374939721-7876-1-git-send-email-armbru@redhat.com>
	<1374939721-7876-2-git-send-email-armbru@redhat.com>
In-Reply-To: <1374939721-7876-2-git-send-email-armbru@redhat.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: Re: [Qemu-devel] [PATCH v2 1/9] tests: QAPI schema parser tests
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Markus Armbruster <armbru@redhat.com>
Cc: akong@redhat.com, qemu-devel@nongnu.org, Anthony Liguori <anthony@codemonkey.ws>, mdroth@linux.vnet.ibm.com

On 07/27/13 17:41, Markus Armbruster wrote:
> The parser handles erroneous input badly.  To be improved shortly.
> 
> Signed-off-by: Markus Armbruster <armbru@redhat.com>

> diff --git a/tests/Makefile b/tests/Makefile
> index cdbb79e..ddb957c 100644
> --- a/tests/Makefile
> +++ b/tests/Makefile

> @@ -233,13 +242,24 @@ check-report.html: check-report.xml
>  check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF)
>  	$<
>  
> +.PHONY: check-tests/test-qapi.py
> +check-tests/test-qapi.py: tests/test-qapi.py
> +
> +.PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
> +$(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
> +	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py <$^ >$*.out 2>$*.err; echo $$? >$*.exit, "  TEST  $*.out")
> +	@diff -q $(SRC_PATH)/$*.out $*.out
> +	@diff -q $(SRC_PATH)/$*.err $*.err
> +	@diff -q $(SRC_PATH)/$*.exit $*.exit
> +

Unfortunately, this doesn't catch errors/differences when someone builds
qemu in the source tree, and runs "make check" there. In that case,
whatever output "test-qapi.py" produces, overwrites the in-tree file,
and the diff commands compare files with themselves.

Also, why is 2/9 a good idea, namely, using "qapi-schema-test.json" as a
test file itself? Whoever adds a new test, for anything that uses the
schema, now has to update the .out file as well. (And, due to the above,
only realizes this burden when someone else tries to make-check his/her
code outside of the tree...)

Thanks,
Laszlo