From mboxrd@z Thu Jan 1 00:00:00 1970 From: Radim =?utf-8?B?S3LEjW3DocWZ?= Subject: Re: [kvm-unit-tests PATCH 1/3] run_tests.sh: reduce return code ambiguity Date: Mon, 21 Dec 2015 17:31:24 +0100 Message-ID: <20151221163124.GA7061@potion.redhat.com> References: <1450383054-9724-1-git-send-email-drjones@redhat.com> <1450383054-9724-2-git-send-email-drjones@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: kvm@vger.kernel.org, pbonzini@redhat.com To: Andrew Jones Return-path: Received: from mx1.redhat.com ([209.132.183.28]:47104 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751199AbbLUQb1 (ORCPT ); Mon, 21 Dec 2015 11:31:27 -0500 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id D85D78E257 for ; Mon, 21 Dec 2015 16:31:27 +0000 (UTC) Content-Disposition: inline In-Reply-To: <1450383054-9724-2-git-send-email-drjones@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: 2015-12-17 14:10-0600, Andrew Jones: > qemu/unittest exit codes are convoluted, causing codes 0 and 1 > to be ambiguous. Here are the possible meanings > > .-----------------------------------------------------------------. > | | 0 | 1 | > |-----------------------------------------------------------------| > | QEMU | did something successfully, | FAILURE | > | | but probably didn't run the | | > | | unittest, OR caught SIGINT, | | > | | SIGHUP, or SIGTERM | | > |-----------------------------------------------------------------| > | unittest | for some reason exited using | SUCCESS | > | | ACPI/PSCI, not with debug-exit | | > .-----------------------------------------------------------------. > > As we can see above, an exit code of zero is even ambiguous for each > row, i.e. QEMU could exit with zero because it successfully completed, > or because it caught a signal. unittest could exit with zero because > it successfully powered-off, or because for some odd reason it powered- > off instead of calling debug-exit. > > And, the most fun is that exit-code == 1 means QEMU failed, but the > unittest succeeded. > > This patch attempts to reduce that ambiguity, by also looking at stderr. Nice. > With it, we have > > 0 - unexpected exit from qemu, or the unittest not using debug-exit. > Consider it a FAILURE > 1 - unittest SUCCESS > < 128 - something failed (could be the unittest, qemu, or a run script) > Check the logs. > >= 128 - signal (signum = code - 128) I think this heuristic should be applied to {arm,x86}/run. run_tests.sh would inherit it and we would finally get reasonable exit values everywhere. The resulting table would look like this: 0 = unit-test passed 77 = unit-test skipped (not implemented yet) 124 = unit-test timeouted (implemented in [3/3]) 127 = qemu returned 0 (debug-exit probably wasn't called) > 128 = exited because of signal $? - 128 * = unit-test failed (Signal 0 is not used, so we could map 128 to mean "debug-exit probably wasn't called", but others might not understand our signal convention. Anyway, it'd be best for us to start at 200, for `case $? in 2??)` ...) > Signed-off-by: Andrew Jones > --- > diff --git a/run_tests.sh b/run_tests.sh > @@ -54,10 +55,32 @@ function run() > > # extra_params in the config file may contain backticks that need to be > # expanded, so use eval to start qemu > - eval $cmdline >> test.log > + errlog=$(mktemp) > + eval $cmdline >> test.log 2> $errlog | [...] | cat $errlog >> test.log This assumes that stderr is always after stdout, eval $cmdline 2>&1 >> test.log | tee $errlog >> test.log has a chance to print lines in wrong order too, but I think it's going to be closer to the original.