Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

[Naresh Kamboju]

On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote:
>
> Currently, after generating run_kselftest.sh, there is no way to choose
> which test we could run. All the tests are listed together and we have
> to run all every time. This patch enhanced the run_kselftest.sh to make
> the tests individually selectable. e.g.
>
>   $ ./run_kselftest.sh -t "bpf size timers"

My test run break on linux next

./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
./run_kselftest.sh: line 1331: `-e -s | --summary )
logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

steps do run:
# run_kselftest.sh file generated by kselftest Makefile and included in tarball
./run_kselftest.sh 2>&1 | tee "${LOGFILE}"

ref:
https://github.com/nareshkamboju/test-definitions/blob/master/automated/linux/kselftest/kselftest.sh#L222

full test run log:
https://lkft.validation.linaro.org/scheduler/job/1786826#L1391

>
> Before the patch:
> ================
>
> $ cat run_kselftest.sh
> \#!/bin/sh
> BASE_DIR=$(realpath $(dirname $0))
> cd $BASE_DIR
> . ./kselftest/runner.sh
> ROOT=$PWD
> if [ "$1" = "--summary" ]; then
>   logfile=$BASE_DIR/output.log
>   cat /dev/null > $logfile
> fi
> [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
> cd android
> run_many        \
>         "run.sh"
> cd $ROOT
> ...<snip>...
> [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
> cd zram
> run_many        \
>         "zram.sh"
> cd $ROOT
>
> After the patch:
> ===============
>
> $ cat run_kselftest.sh
> \#!/bin/sh
> BASE_DIR=$(realpath $(dirname $0))
> . ./kselftest/runner.sh
> TESTS="android ...<snip>... filesystems/binderfs ...<snip>... zram"
>
> run_android()
> {
>         [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
>         cd android
>         run_many        \
>                 "run.sh"
>         cd $ROOT
> }
>
> ...<snip>...
>
> run_filesystems_binderfs()
> {
>         [ -w /dev/kmsg ] && echo "kselftest: Running tests in filesystems/binderfs" >> /dev/kmsg
>         cd filesystems/binderfs
>         run_many        \
>                 "binderfs_test"
>         cd $ROOT
> }
>
> ...<snip>...
>
> run_zram()
> {
>         [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
>         cd zram
>         run_many        \
>                 "zram.sh"
>         cd $ROOT
> }
>
> usage()
> {
>         cat <<EOF
> usage: ${0##*/} OPTS
>         -s | --summary          Only print summary info and put detailed log in output.log
>         -t | --tests            Test name you want to run specifically
>         -h | --help             Show this usage info
> EOF
> }
>
> while true; do
>         case "$1" in
>         -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;
>         -t | --tests ) TESTS=$2; shift 2 ;;
>         -l | --list ) echo $TESTS; exit 0 ;;
>         -h | --help ) usage; exit 0 ;;
>         "" ) break;;
>         * ) usage; exit 1;;
>         esac
> done
>
> cd $BASE_DIR
> ROOT=$PWD
> for folder in $TESTS; do
>         folder=$(echo $folder | tr -s '/-' '_')
>         run_$folder
> done
>
> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com>

Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>

-- 
Linaro LKFT
https://lkft.linaro.org

Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

[Kees Cook]

On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
> On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote:
> >
> > Currently, after generating run_kselftest.sh, there is no way to choose
> > which test we could run. All the tests are listed together and we have
> > to run all every time. This patch enhanced the run_kselftest.sh to make
> > the tests individually selectable. e.g.
> >
> >   $ ./run_kselftest.sh -t "bpf size timers"
> 
> My test run break on linux next
> 
> ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
> ./run_kselftest.sh: line 1331: `-e -s | --summary )
> logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

Yes, please revert this patch. The resulting script is completely
trashed:

BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="seccomp"

run_seccomp()
{
-e      [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
-e      cd seccomp
-en     run_many
        \
-ne             "seccomp_bpf"
        \
-ne             "seccomp_benchmark"

-e      cd $ROOT
}



-- 
Kees Cook

Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

[Shuah Khan]

On 9/25/20 3:16 PM, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
>> On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote:
>>>
>>> Currently, after generating run_kselftest.sh, there is no way to choose
>>> which test we could run. All the tests are listed together and we have
>>> to run all every time. This patch enhanced the run_kselftest.sh to make
>>> the tests individually selectable. e.g.
>>>
>>>    $ ./run_kselftest.sh -t "bpf size timers"
>>
>> My test run break on linux next
>>
>> ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
>> ./run_kselftest.sh: line 1331: `-e -s | --summary )
>> logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
> 
> Yes, please revert this patch. The resulting script is completely
> trashed:
> 
> BASE_DIR=$(realpath $(dirname $0))
> . ./kselftest/runner.sh
> TESTS="seccomp"
> 
> run_seccomp()
> {
> -e      [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
> -e      cd seccomp
> -en     run_many
>          \
> -ne             "seccomp_bpf"
>          \
> -ne             "seccomp_benchmark"
> 
> -e      cd $ROOT
> }
> 
> 
> 

Will do.

thanks,
-- Shuah

Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

[Shuah Khan]

On 9/25/20 3:16 PM, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
>> On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote:
>>>
>>> Currently, after generating run_kselftest.sh, there is no way to choose
>>> which test we could run. All the tests are listed together and we have
>>> to run all every time. This patch enhanced the run_kselftest.sh to make
>>> the tests individually selectable. e.g.
>>>
>>>    $ ./run_kselftest.sh -t "bpf size timers"
>>
>> My test run break on linux next
>>
>> ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
>> ./run_kselftest.sh: line 1331: `-e -s | --summary )
>> logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
> 
> Yes, please revert this patch. The resulting script is completely
> trashed:
> 

Thank you both. Now reverted.

thanks,
-- Shuah

Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

[Kees Cook]

On Fri, Sep 25, 2020 at 05:06:02PM -0600, Shuah Khan wrote:
> On 9/25/20 3:16 PM, Kees Cook wrote:
> > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
> > > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote:
> > > > 
> > > > Currently, after generating run_kselftest.sh, there is no way to choose
> > > > which test we could run. All the tests are listed together and we have
> > > > to run all every time. This patch enhanced the run_kselftest.sh to make
> > > > the tests individually selectable. e.g.
> > > > 
> > > >    $ ./run_kselftest.sh -t "bpf size timers"
> > > 
> > > My test run break on linux next
> > > 
> > > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
> > > ./run_kselftest.sh: line 1331: `-e -s | --summary )
> > > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
> > 
> > Yes, please revert this patch. The resulting script is completely
> > trashed:
> > 
> 
> Thank you both. Now reverted.

I've sent an alternative that I think should do nicely. It will work
well with LAVA as well.

-- 
Kees Cook

Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

[Hangbin Liu]

On Fri, Sep 25, 2020 at 02:16:14PM -0700, Kees Cook wrote:
> On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
> > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote:
> > >
> > > Currently, after generating run_kselftest.sh, there is no way to choose
> > > which test we could run. All the tests are listed together and we have
> > > to run all every time. This patch enhanced the run_kselftest.sh to make
> > > the tests individually selectable. e.g.
> > >
> > >   $ ./run_kselftest.sh -t "bpf size timers"
> > 
> > My test run break on linux next
> > 
> > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
> > ./run_kselftest.sh: line 1331: `-e -s | --summary )
> > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
> 
> Yes, please revert this patch. The resulting script is completely
> trashed:
> 
> BASE_DIR=$(realpath $(dirname $0))
> . ./kselftest/runner.sh
> TESTS="seccomp"
> 
> run_seccomp()
> {
> -e      [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
> -e      cd seccomp
> -en     run_many
>         \
> -ne             "seccomp_bpf"
>         \
> -ne             "seccomp_benchmark"
> 
> -e      cd $ROOT
> }

I'm really sorry to make this trouble. And I'm OK to revert the patch.
I just a little wondering how do you generate this script.
I tested with 'make -C tools/testing/selftests gen_tar FORMAT=.xz' before
post the patch and all looks good to me.

```

run_seccomp()
{
        [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
        cd seccomp
        run_many        \
                "seccomp_bpf"   \
                "seccomp_benchmark"
        cd $ROOT
}

```

Thanks
Hangbin

Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

[Kees Cook]

On Sun, Sep 27, 2020 at 09:53:34AM +0800, Hangbin Liu wrote:
> On Fri, Sep 25, 2020 at 02:16:14PM -0700, Kees Cook wrote:
> > On Fri, Sep 25, 2020 at 01:51:53PM +0530, Naresh Kamboju wrote:
> > > On Mon, 14 Sep 2020 at 07:53, Hangbin Liu <liuhangbin@gmail.com> wrote:
> > > >
> > > > Currently, after generating run_kselftest.sh, there is no way to choose
> > > > which test we could run. All the tests are listed together and we have
> > > > to run all every time. This patch enhanced the run_kselftest.sh to make
> > > > the tests individually selectable. e.g.
> > > >
> > > >   $ ./run_kselftest.sh -t "bpf size timers"
> > > 
> > > My test run break on linux next
> > > 
> > > ./run_kselftest.sh: line 1331: syntax error near unexpected token `)'
> > > ./run_kselftest.sh: line 1331: `-e -s | --summary )
> > > logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'
> > 
> > Yes, please revert this patch. The resulting script is completely
> > trashed:
> > 
> > BASE_DIR=$(realpath $(dirname $0))
> > . ./kselftest/runner.sh
> > TESTS="seccomp"
> > 
> > run_seccomp()
> > {
> > -e      [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
> > -e      cd seccomp
> > -en     run_many
> >         \
> > -ne             "seccomp_bpf"
> >         \
> > -ne             "seccomp_benchmark"
> > 
> > -e      cd $ROOT
> > }
> 
> I'm really sorry to make this trouble. And I'm OK to revert the patch.
> I just a little wondering how do you generate this script.

This issue is with which shell is used. I suspect your /bin/sh is full
/bin/bash, where as Naresh's, the CI's, and mine are /bin/dash (which
lacks "-e" support for the built-in "echo").

-- 
Kees Cook

Re: [PATCHv5 kselftest next] selftests/run_kselftest.sh: make each test individually selectable

[Hangbin Liu]

On Mon, Sep 28, 2020 at 01:06:15PM -0700, Kees Cook wrote:
> > I'm really sorry to make this trouble. And I'm OK to revert the patch.
> > I just a little wondering how do you generate this script.
> 
> This issue is with which shell is used. I suspect your /bin/sh is full
> /bin/bash, where as Naresh's, the CI's, and mine are /bin/dash (which
> lacks "-e" support for the built-in "echo").

Ah, got it. Thanks for your explanation.

Regards
Hangbin