From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Jones Subject: Re: [kvm-unit-tests PATCH 1/3] run_tests.sh: reduce return code ambiguity Date: Mon, 21 Dec 2015 13:35:10 -0600 Message-ID: <20151221193510.GA5123@hawk.localdomain> References: <1450383054-9724-1-git-send-email-drjones@redhat.com> <1450383054-9724-2-git-send-email-drjones@redhat.com> <20151221163124.GA7061@potion.redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: kvm@vger.kernel.org, pbonzini@redhat.com To: Radim =?utf-8?B?S3LEjW3DocWZ?= Return-path: Received: from mx1.redhat.com ([209.132.183.28]:34679 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751291AbbLUTfQ (ORCPT ); Mon, 21 Dec 2015 14:35:16 -0500 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 381468F294 for ; Mon, 21 Dec 2015 19:35:16 +0000 (UTC) Content-Disposition: inline In-Reply-To: <20151221163124.GA7061@potion.redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On Mon, Dec 21, 2015 at 05:31:24PM +0100, Radim Kr=C4=8Dm=C3=A1=C5=99 w= rote: > 2015-12-17 14:10-0600, Andrew Jones: > > qemu/unittest exit codes are convoluted, causing codes 0 and 1 > > to be ambiguous. Here are the possible meanings > >=20 > > .-----------------------------------------------------------------= =2E > > | | 0 | 1 = | > > |-----------------------------------------------------------------= | > > | QEMU | did something successfully, | FAILURE = | > > | | but probably didn't run the | = | > > | | unittest, OR caught SIGINT, | = | > > | | SIGHUP, or SIGTERM | = | > > |-----------------------------------------------------------------= | > > | unittest | for some reason exited using | SUCCESS = | > > | | ACPI/PSCI, not with debug-exit | = | > > .-----------------------------------------------------------------= =2E > >=20 > > As we can see above, an exit code of zero is even ambiguous for eac= h > > row, i.e. QEMU could exit with zero because it successfully complet= ed, > > or because it caught a signal. unittest could exit with zero becaus= e > > it successfully powered-off, or because for some odd reason it powe= red- > > off instead of calling debug-exit. > >=20 > > And, the most fun is that exit-code =3D=3D 1 means QEMU failed, but= the > > unittest succeeded. > >=20 > > This patch attempts to reduce that ambiguity, by also looking at st= derr. >=20 > Nice. >=20 > > With it, we have > >=20 > > 0 - unexpected exit from qemu, or the unittest not using debu= g-exit. > > Consider it a FAILURE > > 1 - unittest SUCCESS > > < 128 - something failed (could be the unittest, qemu, or a run s= cript) > > Check the logs. > > >=3D 128 - signal (signum =3D code - 128) >=20 > I think this heuristic should be applied to {arm,x86}/run. > run_tests.sh would inherit it and we would finally get reasonable exi= t > values everywhere. Good idea. We can add the table to a scripts/functions.bash function an= d use it everywhere. >=20 > The resulting table would look like this: >=20 > 0 =3D unit-test passed > 77 =3D unit-test skipped (not implemented yet) > 124 =3D unit-test timeouted (implemented in [3/3]) > 127 =3D qemu returned 0 (debug-exit probably wasn't called) We already use 127 for abort(), called from a unit test, see lib/abort.c. I guess we can use 126 for "debug-exit probably wasn't called". We should also add a (unit test called abort) message for 127. > > 128 =3D exited because of signal $? - 128 > * =3D unit-test failed >=20 > (Signal 0 is not used, so we could map 128 to mean "debug-exit probab= ly > wasn't called", but others might not understand our signal conventio= n. I think we want 128 to be the beginning of signal space, which goes all the way up to 255, in order to allow exit code masking to work. > Anyway, it'd be best for us to start at 200, for `case $? in 2??)` .= =2E.) Start what at 200? I think we have everything covered above. The mappin= g looks like this 0 =3D success 1-63 =3D unit test failure code 64-127 =3D test suite failure code 128-255 =3D signal which sounds good to me. >=20 > > Signed-off-by: Andrew Jones > > --- > > diff --git a/run_tests.sh b/run_tests.sh > > @@ -54,10 +55,32 @@ function run() > > =20 > > # extra_params in the config file may contain backticks that n= eed to be > > # expanded, so use eval to start qemu > > - eval $cmdline >> test.log > > + errlog=3D$(mktemp) > > + eval $cmdline >> test.log 2> $errlog > | [...] > | cat $errlog >> test.log >=20 > This assumes that stderr is always after stdout, True. I'm not sure that matters when the unit test, which only uses std= out will always output stuff serially with qemu, which could output a mix. = But your version below is fine by me if we want to pick up the need for the pipe and tee. >=20 > eval $cmdline 2>&1 >> test.log | tee $errlog >> test.log >=20 > has a chance to print lines in wrong order too, but I think it's goin= g > to be closer to the original. I'll play with it and send a v2 soon. Thanks, drew > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html