[LTP] Failing test on failing shell commands

[LTP] Failing test on failing shell commands

[Krzysztof Kozlowski]

Hi all,

I have a failing memcg_use_hierarchy_test (in
testcases/kernel/controllers/memcg/functional/) which hits unexpected
OOM on one of earlier commands, already in the test:

 18         echo 1 > memory.use_hierarchy
 19         echo $PAGESIZE > memory.limit_in_bytes
 20 
 21         mkdir subgroup  
mkdir: cannot create directory ?subgroup?: Cannot allocate memory
/home/ubuntu/ltp-install/testcases/bin/memcg_use_hierarchy_test.sh: 22: cd: can't cd to subgroup

The command did not fail, as there is no "set -e" and test fails later.
However there could be a case where such failure does not trigger later
issue and test is a false-positive.

What is the practice for LTP shell tests to handle kind of preparation
commands? Maybe this should be something like:

--------

diff --git a/testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh
index 8be342499ece..f8dd8946f80b 100755
--- a/testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh
+++ b/testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh
@@ -18,8 +18,8 @@ test1()
        echo 1 > memory.use_hierarchy
        echo $PAGESIZE > memory.limit_in_bytes
 
-       mkdir subgroup
-       cd subgroup
+       EXPECT_PASS mkdir subgroup
+       EXPECT_PASS cd subgroup
        test_proc_kill $((PAGESIZE * 3)) "--mmap-lock1" $((PAGESIZE * 2)) 0
 
        cd ..

Best regards,
Krzysztof

[LTP] Failing test on failing shell commands

[Cyril Hrubis]

Hi!
> I have a failing memcg_use_hierarchy_test (in
> testcases/kernel/controllers/memcg/functional/) which hits unexpected
> OOM on one of earlier commands, already in the test:
> 
>  18         echo 1 > memory.use_hierarchy
>  19         echo $PAGESIZE > memory.limit_in_bytes
>  20 
>  21         mkdir subgroup  
> mkdir: cannot create directory ???subgroup???: Cannot allocate memory
> /home/ubuntu/ltp-install/testcases/bin/memcg_use_hierarchy_test.sh: 22: cd: can't cd to subgroup
> 
> The command did not fail, as there is no "set -e" and test fails later.
> However there could be a case where such failure does not trigger later
> issue and test is a false-positive.
> 
> What is the practice for LTP shell tests to handle kind of preparation
> commands? Maybe this should be something like:

We do have ROD (as a shorthand for "run or die") in the shell test
library, so this really should be:

	ROD mkdir subgroup

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] Failing test on failing shell commands

[Krzysztof Kozlowski]

On 13/07/2021 13:55, Cyril Hrubis wrote:
> Hi!
>> I have a failing memcg_use_hierarchy_test (in
>> testcases/kernel/controllers/memcg/functional/) which hits unexpected
>> OOM on one of earlier commands, already in the test:
>>
>>  18         echo 1 > memory.use_hierarchy
>>  19         echo $PAGESIZE > memory.limit_in_bytes
>>  20 
>>  21         mkdir subgroup  
>> mkdir: cannot create directory ???subgroup???: Cannot allocate memory
>> /home/ubuntu/ltp-install/testcases/bin/memcg_use_hierarchy_test.sh: 22: cd: can't cd to subgroup
>>
>> The command did not fail, as there is no "set -e" and test fails later.
>> However there could be a case where such failure does not trigger later
>> issue and test is a false-positive.
>>
>> What is the practice for LTP shell tests to handle kind of preparation
>> commands? Maybe this should be something like:
> 
> We do have ROD (as a shorthand for "run or die") in the shell test
> library, so this really should be:
> 
> 	ROD mkdir subgroup

Thanks. This will however fail entire suite (memcg_use_hierarchy_test)
and other tests won't get executed. The failure could be only local, so
other tests are still meaningful.


Best regards,
Krzysztof

[LTP] Failing test on failing shell commands

[Cyril Hrubis]

Hi!
> > We do have ROD (as a shorthand for "run or die") in the shell test
> > library, so this really should be:
> > 
> > 	ROD mkdir subgroup
> 
> Thanks. This will however fail entire suite (memcg_use_hierarchy_test)
> and other tests won't get executed. The failure could be only local, so
> other tests are still meaningful.

If you want to exit the subtest you really have to handle the return
values yourself, report the error, and return from the test function.

The question is which status to report. The TBROK status means that test
has failed setup phase and there is no point to continue, so this
probably should map to tst_res TWARN "...".

Another solution would be splitting the test into three seprate tests
so that we can use ROD.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] Failing test on failing shell commands

[Krzysztof Kozlowski]

On 13/07/2021 13:56, Krzysztof Kozlowski wrote:
> 
> Hi all,
> 
> I have a failing memcg_use_hierarchy_test (in
> testcases/kernel/controllers/memcg/functional/) which hits unexpected
> OOM on one of earlier commands, already in the test:
> 
>  18         echo 1 > memory.use_hierarchy
>  19         echo $PAGESIZE > memory.limit_in_bytes
>  20 
>  21         mkdir subgroup  
> mkdir: cannot create directory ?subgroup?: Cannot allocate memory
> /home/ubuntu/ltp-install/testcases/bin/memcg_use_hierarchy_test.sh: 22: cd: can't cd to subgroup
> 
> The command did not fail, as there is no "set -e" and test fails later.

Ah, I wanted to say:
"The command did not fail entire test, as there is no "set -e" and test
fails later."

> However there could be a case where such failure does not trigger later
> issue and test is a false-positive.
> 
> What is the practice for LTP shell tests to handle kind of preparation
> commands? Maybe this should be something like:
> 
> --------
> 
> diff --git a/testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh b/testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh
> index 8be342499ece..f8dd8946f80b 100755
> --- a/testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh
> +++ b/testcases/kernel/controllers/memcg/functional/memcg_use_hierarchy_test.sh
> @@ -18,8 +18,8 @@ test1()
>         echo 1 > memory.use_hierarchy
>         echo $PAGESIZE > memory.limit_in_bytes
>  
> -       mkdir subgroup
> -       cd subgroup
> +       EXPECT_PASS mkdir subgroup
> +       EXPECT_PASS cd subgroup
>         test_proc_kill $((PAGESIZE * 3)) "--mmap-lock1" $((PAGESIZE * 2)) 0
>  
>         cd ..
> 
> Best regards,
> Krzysztof
> 


Best regards,
Krzysztof