[PATCH nft] tests: run-tests.sh: ensure non-zero exit when $failed != 0

[PATCH nft] tests: run-tests.sh: ensure non-zero exit when $failed != 0

[Štěpán Němec]

POSIX [1] does not specify the behavior of `exit' with arguments
outside the 0-255 range, but what generally (bash, dash, zsh, OpenBSD
ksh, busybox) seems to happen is the shell exiting with status & 255
[2], which results in zero exit for certain non-zero arguments.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#exit
[2] https://git.savannah.gnu.org/cgit/bash.git/tree/builtins/common.c#n579

Fixes: 0c6592420586 ("tests: fix return codes")
Signed-off-by: Štěpán Němec <snemec@redhat.com>
---
 tests/build/run-tests.sh | 2 +-
 tests/shell/run-tests.sh | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/build/run-tests.sh b/tests/build/run-tests.sh
index 9ce93a8ed381..f78cc9019a30 100755
--- a/tests/build/run-tests.sh
+++ b/tests/build/run-tests.sh
@@ -52,4 +52,4 @@ done
 rm -rf $tmpdir
 
 echo "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
-exit $failed
+[ "$failed" -eq 0 ]
diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh
index 349ec6cb1b16..f77d850ef285 100755
--- a/tests/shell/run-tests.sh
+++ b/tests/shell/run-tests.sh
@@ -160,4 +160,4 @@ echo ""
 msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
 
 kernel_cleanup
-exit $failed
+[ "$failed" -eq 0 ]

base-commit: 2139913694a9850c9160920b2c638aac4828f9bb
-- 
2.33.1

Re: [PATCH nft] tests: run-tests.sh: ensure non-zero exit when $failed != 0

[Phil Sutter]

Hi,

On Wed, Oct 20, 2021 at 02:44:09PM +0200, Štěpán Němec wrote:
> POSIX [1] does not specify the behavior of `exit' with arguments
> outside the 0-255 range, but what generally (bash, dash, zsh, OpenBSD
> ksh, busybox) seems to happen is the shell exiting with status & 255
> [2], which results in zero exit for certain non-zero arguments.

Standards aside, failed=256 is an actual bug:

| % bash -c "exit 255"; echo $?
| 255
| % bash -c "exit 256"; echo $?
| 0
| % bash -c "exit 257"; echo $?
| 1

Thanks, Phil

Re: [PATCH nft] tests: run-tests.sh: ensure non-zero exit when $failed != 0

[Pablo Neira Ayuso]

Hi Phil,

On Wed, Oct 20, 2021 at 05:06:41PM +0200, Phil Sutter wrote:
> Hi,
> 
> On Wed, Oct 20, 2021 at 02:44:09PM +0200, Štěpán Němec wrote:
> > POSIX [1] does not specify the behavior of `exit' with arguments
> > outside the 0-255 range, but what generally (bash, dash, zsh, OpenBSD
> > ksh, busybox) seems to happen is the shell exiting with status & 255
> > [2], which results in zero exit for certain non-zero arguments.
> 
> Standards aside, failed=256 is an actual bug:
> 
> | % bash -c "exit 255"; echo $?
> | 255
> | % bash -c "exit 256"; echo $?
> | 0
> | % bash -c "exit 257"; echo $?
> | 1

This is extra information you provided here for the commit message for
completion?

Thanks.

Re: [PATCH nft] tests: run-tests.sh: ensure non-zero exit when $failed != 0

[Phil Sutter]

Hi Pablo,

On Wed, Oct 27, 2021 at 11:09:23AM +0200, Pablo Neira Ayuso wrote:
> On Wed, Oct 20, 2021 at 05:06:41PM +0200, Phil Sutter wrote:
> > On Wed, Oct 20, 2021 at 02:44:09PM +0200, Štěpán Němec wrote:
> > > POSIX [1] does not specify the behavior of `exit' with arguments
> > > outside the 0-255 range, but what generally (bash, dash, zsh, OpenBSD
> > > ksh, busybox) seems to happen is the shell exiting with status & 255
> > > [2], which results in zero exit for certain non-zero arguments.
> > 
> > Standards aside, failed=256 is an actual bug:
> > 
> > | % bash -c "exit 255"; echo $?
> > | 255
> > | % bash -c "exit 256"; echo $?
> > | 0
> > | % bash -c "exit 257"; echo $?
> > | 1
> 
> This is extra information you provided here for the commit message for
> completion?

No need to extend the commit message IMO. I was just curious and played
a bit with exit values in bash. So although unlikely, the unpatched code
indeed confuses a result of 256 errors for a pass. :)

Cheers, Phil

Re: [PATCH nft] tests: run-tests.sh: ensure non-zero exit when $failed != 0

[Pablo Neira Ayuso]

On Tue, Nov 02, 2021 at 09:31:15PM +0100, Phil Sutter wrote:
> Hi Pablo,
> 
> On Wed, Oct 27, 2021 at 11:09:23AM +0200, Pablo Neira Ayuso wrote:
> > On Wed, Oct 20, 2021 at 05:06:41PM +0200, Phil Sutter wrote:
> > > On Wed, Oct 20, 2021 at 02:44:09PM +0200, Štěpán Němec wrote:
> > > > POSIX [1] does not specify the behavior of `exit' with arguments
> > > > outside the 0-255 range, but what generally (bash, dash, zsh, OpenBSD
> > > > ksh, busybox) seems to happen is the shell exiting with status & 255
> > > > [2], which results in zero exit for certain non-zero arguments.
> > > 
> > > Standards aside, failed=256 is an actual bug:
> > > 
> > > | % bash -c "exit 255"; echo $?
> > > | 255
> > > | % bash -c "exit 256"; echo $?
> > > | 0
> > > | % bash -c "exit 257"; echo $?
> > > | 1
> > 
> > This is extra information you provided here for the commit message for
> > completion?
> 
> No need to extend the commit message IMO. I was just curious and played
> a bit with exit values in bash. So although unlikely, the unpatched code
> indeed confuses a result of 256 errors for a pass. :)

OK, then please go push out this patch if you're fine with it.

Re: [PATCH nft] tests: run-tests.sh: ensure non-zero exit when $failed != 0

[Phil Sutter]

On Tue, Nov 02, 2021 at 09:44:47PM +0100, Pablo Neira Ayuso wrote:
> On Tue, Nov 02, 2021 at 09:31:15PM +0100, Phil Sutter wrote:
> > Hi Pablo,
> > 
> > On Wed, Oct 27, 2021 at 11:09:23AM +0200, Pablo Neira Ayuso wrote:
> > > On Wed, Oct 20, 2021 at 05:06:41PM +0200, Phil Sutter wrote:
> > > > On Wed, Oct 20, 2021 at 02:44:09PM +0200, Štěpán Němec wrote:
> > > > > POSIX [1] does not specify the behavior of `exit' with arguments
> > > > > outside the 0-255 range, but what generally (bash, dash, zsh, OpenBSD
> > > > > ksh, busybox) seems to happen is the shell exiting with status & 255
> > > > > [2], which results in zero exit for certain non-zero arguments.
> > > > 
> > > > Standards aside, failed=256 is an actual bug:
> > > > 
> > > > | % bash -c "exit 255"; echo $?
> > > > | 255
> > > > | % bash -c "exit 256"; echo $?
> > > > | 0
> > > > | % bash -c "exit 257"; echo $?
> > > > | 1
> > > 
> > > This is extra information you provided here for the commit message for
> > > completion?
> > 
> > No need to extend the commit message IMO. I was just curious and played
> > a bit with exit values in bash. So although unlikely, the unpatched code
> > indeed confuses a result of 256 errors for a pass. :)
> 
> OK, then please go push out this patch if you're fine with it.

DONE. Sorry for the confusion.

Cheers, Phil