[LTP] [PATCH 0/2] Update test timeouts in an automated way

public inbox for ltp@lists.linux.it
 help / color / mirror / Atom feed

* [LTP] [PATCH 0/2] Update test timeouts in an automated way
@ 2025-01-21 12:34 Cyril Hrubis
  2025-01-21 12:34 ` [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts Cyril Hrubis
  2025-01-21 12:34 ` [LTP] [PATCH 2/2] syscalls: Update test timeouts Cyril Hrubis
  0 siblings, 2 replies; 8+ messages in thread
From: Cyril Hrubis @ 2025-01-21 12:34 UTC (permalink / raw)
  To: ltp

First patch adds a script that can parse kirk test results, calculate
timeouts and patch tests with the newly calculated timeouts.

Second patch updates tests timeouts using this script with a results
captured on RPi zero.

Cyril Hrubis (2):
  scripts: Add simple script for calculating timeouts
  syscalls: Update test timeouts

 scripts/calc_timeouts.py                      | 133 ++++++++++++++++++
 testcases/kernel/syscalls/access/access01.c   |   1 +
 testcases/kernel/syscalls/add_key/add_key05.c |   1 +
 testcases/kernel/syscalls/alarm/alarm05.c     |   1 +
 testcases/kernel/syscalls/alarm/alarm06.c     |   1 +
 testcases/kernel/syscalls/alarm/alarm07.c     |   1 +
 testcases/kernel/syscalls/bind/bind04.c       |   1 +
 testcases/kernel/syscalls/bpf/bpf_prog05.c    |   1 +
 testcases/kernel/syscalls/bpf/bpf_prog06.c    |   1 +
 testcases/kernel/syscalls/bpf/bpf_prog07.c    |   1 +
 .../kernel/syscalls/cachestat/cachestat01.c   |   1 +
 .../kernel/syscalls/cachestat/cachestat04.c   |   1 +
 testcases/kernel/syscalls/chdir/chdir01.c     |   1 +
 .../kernel/syscalls/clock_gettime/leapsec01.c |   1 +
 .../clock_nanosleep/clock_nanosleep01.c       |   1 +
 .../syscalls/clock_settime/clock_settime03.c  |   1 +
 .../syscalls/close_range/close_range01.c      |   1 +
 testcases/kernel/syscalls/connect/connect02.c |   1 +
 testcases/kernel/syscalls/creat/creat05.c     |   1 +
 testcases/kernel/syscalls/creat/creat09.c     |   1 +
 testcases/kernel/syscalls/execve/execve05.c   |   1 +
 .../kernel/syscalls/execveat/execveat03.c     |   1 +
 .../kernel/syscalls/fallocate/fallocate04.c   |   1 +
 .../kernel/syscalls/fallocate/fallocate05.c   |   1 +
 .../kernel/syscalls/fallocate/fallocate06.c   |   2 +-
 .../kernel/syscalls/fanotify/fanotify01.c     |   1 +
 .../kernel/syscalls/fanotify/fanotify03.c     |   1 +
 .../kernel/syscalls/fanotify/fanotify05.c     |   1 +
 .../kernel/syscalls/fanotify/fanotify06.c     |   1 +
 .../kernel/syscalls/fanotify/fanotify09.c     |   1 +
 .../kernel/syscalls/fanotify/fanotify10.c     |   1 +
 .../kernel/syscalls/fchmodat2/fchmodat2_01.c  |   1 +
 testcases/kernel/syscalls/fcntl/fcntl14.c     |   1 +
 testcases/kernel/syscalls/fcntl/fcntl36.c     |   1 +
 .../kernel/syscalls/fdatasync/fdatasync03.c   |   1 +
 .../kernel/syscalls/fgetxattr/fgetxattr01.c   |   1 +
 .../syscalls/fremovexattr/fremovexattr01.c    |   1 +
 .../syscalls/fremovexattr/fremovexattr02.c    |   1 +
 .../kernel/syscalls/fsconfig/fsconfig01.c     |   1 +
 .../kernel/syscalls/fsconfig/fsconfig03.c     |   1 +
 .../kernel/syscalls/fsetxattr/fsetxattr01.c   |   1 +
 testcases/kernel/syscalls/fsmount/fsmount01.c |   1 +
 testcases/kernel/syscalls/fsmount/fsmount02.c |   1 +
 testcases/kernel/syscalls/fsopen/fsopen01.c   |   1 +
 testcases/kernel/syscalls/fspick/fspick01.c   |   1 +
 testcases/kernel/syscalls/fspick/fspick02.c   |   1 +
 testcases/kernel/syscalls/fstatfs/fstatfs01.c |   1 +
 testcases/kernel/syscalls/fsync/fsync01.c     |   1 +
 testcases/kernel/syscalls/fsync/fsync04.c     |   1 +
 testcases/kernel/syscalls/getpid/getpid01.c   |   1 +
 .../kernel/syscalls/getxattr/getxattr02.c     |   1 +
 .../kernel/syscalls/getxattr/getxattr03.c     |   1 +
 testcases/kernel/syscalls/inotify/inotify03.c |   1 +
 testcases/kernel/syscalls/inotify/inotify05.c |   1 +
 testcases/kernel/syscalls/inotify/inotify07.c |   1 +
 testcases/kernel/syscalls/inotify/inotify08.c |   1 +
 testcases/kernel/syscalls/inotify/inotify11.c |   1 +
 testcases/kernel/syscalls/ioctl/ioctl02.c     |   1 +
 testcases/kernel/syscalls/ioctl/ioctl04.c     |   1 +
 testcases/kernel/syscalls/ioctl/ioctl08.c     |   1 +
 testcases/kernel/syscalls/ioctl/ioctl09.c     |   1 +
 .../kernel/syscalls/ioctl/ioctl_ficlone02.c   |   1 +
 .../kernel/syscalls/ioctl/ioctl_ficlone03.c   |   1 +
 .../syscalls/ioctl/ioctl_ficlonerange01.c     |   1 +
 .../syscalls/ioctl/ioctl_ficlonerange02.c     |   1 +
 .../kernel/syscalls/ioctl/ioctl_loop01.c      |   1 +
 66 files changed, 198 insertions(+), 1 deletion(-)
 create mode 100755 scripts/calc_timeouts.py

-- 
2.45.2


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts
  2025-01-21 12:34 [LTP] [PATCH 0/2] Update test timeouts in an automated way Cyril Hrubis
@ 2025-01-21 12:34 ` Cyril Hrubis
  2025-01-21 13:12   ` Andrea Cervesato via ltp
  2025-01-21 12:34 ` [LTP] [PATCH 2/2] syscalls: Update test timeouts Cyril Hrubis
  1 sibling, 1 reply; 8+ messages in thread
From: Cyril Hrubis @ 2025-01-21 12:34 UTC (permalink / raw)
  To: ltp

This script parses JSON results from kirk and LTP metadata in order
calculate timeouts for tests based on the result file and can even patch
tests automatically.

The script does:

- Take the results and pick all tests that run for longer than 0.5s.
  Multiplies the time with a constant (currently 1.2) to get a suggested
  timeout.

- Exclude tests that have runtime defined since these are controller
  by the runtime (that filters out all fuzzy sync tests).

  There is a special case for timer tests that define runtime only
  dynamically in the timer library code. This should be possibly fixed
  with special value for the .runtime in tst_test. E.g.
  TST_RUNTIME_DYNAMIC for tests that only set runtime in the setup.

- Normalize the timeout per a single filesystem run if test is running for
  more than one filesystem.

- Tests that do not have a metadata record are old library tests which
  which cannot be patched but are printed in a separate table if we
  request a table to be printed.

- If patching option is selected tests are update with newly calculated
  timeout. By default we only increase timeouts but that can be
  overrided with the -o option.

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 scripts/calc_timeouts.py | 133 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)
 create mode 100755 scripts/calc_timeouts.py

diff --git a/scripts/calc_timeouts.py b/scripts/calc_timeouts.py
new file mode 100755
index 000000000..c69ab8f57
--- /dev/null
+++ b/scripts/calc_timeouts.py
@@ -0,0 +1,133 @@
+#!/usr/bin/python
+import re
+import json
+import getopt
+import sys
+from os import system
+
+# Top level directory path
+top_dir = '../'
+
+# The test runtime is multiplied by this to get a timeout
+timeout_mul = 1.2
+
+def patch(fname, new_timeout, patch_override):
+
+    orig_timeout = None
+    file_path = top_dir + fname
+
+    with open(file_path, 'r') as c_source:
+        for line in c_source:
+            timeout = re.search(r'\s*.timeout\s*=\s*(\d+).', line)
+            if timeout:
+                orig_timeout = int(timeout.group(1))
+
+    if orig_timeout:
+        if orig_timeout < new_timeout or patch_override:
+            print("CHANGE %s timeout %i -> %i" % (fname, orig_timeout, new_timeout))
+            system("sed -i 's/\\.timeout = [0-9]*/\\.timeout = " + str(new_timeout) + "/' " + file_path)
+        else:
+            print("KEEP   %s timeout %i (new %i)" % (fname, orig_timeout, new_timeout))
+    else:
+        print("ADD    %s timeout %i" % (fname, new_timeout))
+        system("sed -i '/static struct tst_test test = {/a\\\\t.timeout = " + str(new_timeout) + ",' " + file_path)
+
+def patch_all(timeouts, patch_override):
+    for timeout in timeouts:
+        if timeout[3]:
+            patch(timeout[3], timeout[1], patch_override)
+
+def print_table(timeouts):
+    timeouts.sort(key=lambda x: x[1], reverse=True)
+
+    total = 0;
+
+    print("Old library tests\n-----------------\n");
+    for timeout in timeouts:
+        if not timeout[2]:
+            print("%-30s %i" % (timeout[0], timeout[1]))
+            total+=1
+
+    print("\n\t%i tests in total" % total)
+
+    total = 0;
+
+    print("\nNew library tests\n-----------------\n");
+    for timeout in timeouts:
+        if timeout[2]:
+            print("%-30s %i" % (timeout[0], timeout[1]))
+            total+=1
+
+    print("\n\t%i tests in total" % total)
+
+def parse_data(results_path):
+    timeouts = []
+
+    with open(results_path, 'r') as file:
+        results = json.load(file)
+
+    with open(top_dir + 'metadata/ltp.json', 'r') as file:
+        metadata = json.load(file)
+
+    for test in results['results']:
+        name = test['test_fqn']
+        duration = test['test']['duration']
+        # If test runs for all_filesystems normalize the runtime per a single filesystem
+        filesystems = max(1, test['test']['log'].count('TINFO: Formatting /'))
+        # Check if test is new library test
+        test_is_newlib = name in metadata['tests']
+        # Store test file path
+        path = None if not test_is_newlib else metadata['tests'][name]['fname']
+        # Filter out tests with runtime
+        test_has_runtime = False if not test_is_newlib else 'runtime' in metadata['tests'][name]
+        # Timer tests define runtime dynamically in timer library
+        if test_is_newlib and 'sample' in metadata['tests'][name]:
+            test_has_runtime = True
+        # Select tests that does not have runtime and are executed for longer time
+        if not test_has_runtime and duration >= 0.5:
+            timeouts.append((name, int(timeout_mul * duration/filesystems + 0.5), test_is_newlib, path))
+
+    return timeouts
+
+def print_help():
+    print('calc_timeouts.py [OPTION] [RESULT].json')
+    print('\t-h prints this help')
+    print('\t-o override test timeouts, by default timeouts are only increased')
+    print('\t-p patch testcases with updated timeouts')
+    print('\t-t prints table of tests with suggested timeouts')
+
+try:
+    opts, args = getopt.getopt(sys.argv[1:], "hopt")
+except:
+    print_help()
+    sys.exit(1)
+
+opt_print_table = False
+opt_patch_tests = False
+opt_patch_override = False
+
+for opt,arg in opts:
+    if opt == '-h':
+        print_help()
+        sys.exit(0)
+    if opt == '-o':
+        opt_patch_override = True
+    if opt == '-p':
+        opt_patch_tests = True
+    if opt == '-t':
+        opt_print_table = True
+
+if not opt_print_table and not opt_patch_tests:
+    print("No action selected!\n")
+    print_help()
+    sys.exit(1)
+
+results = args[0] if args else 'results.json'
+
+timeouts = parse_data(results)
+
+if opt_print_table:
+    print_table(timeouts)
+
+if opt_patch_tests:
+    patch_all(timeouts, opt_patch_override)
-- 
2.45.2


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts
  2025-01-21 12:34 ` [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts Cyril Hrubis
@ 2025-01-21 13:12   ` Andrea Cervesato via ltp
  2025-01-21 14:26     ` Cyril Hrubis
  0 siblings, 1 reply; 8+ messages in thread
From: Andrea Cervesato via ltp @ 2025-01-21 13:12 UTC (permalink / raw)
  To: Cyril Hrubis, ltp

Hi!

I like the general idea, a couple of comments on the python code.

On 1/21/25 13:34, Cyril Hrubis wrote:
> This script parses JSON results from kirk and LTP metadata in order
> calculate timeouts for tests based on the result file and can even patch
> tests automatically.
>
> The script does:
>
> - Take the results and pick all tests that run for longer than 0.5s.
>    Multiplies the time with a constant (currently 1.2) to get a suggested
>    timeout.
>
> - Exclude tests that have runtime defined since these are controller
>    by the runtime (that filters out all fuzzy sync tests).
>
>    There is a special case for timer tests that define runtime only
>    dynamically in the timer library code. This should be possibly fixed
>    with special value for the .runtime in tst_test. E.g.
>    TST_RUNTIME_DYNAMIC for tests that only set runtime in the setup.
>
> - Normalize the timeout per a single filesystem run if test is running for
>    more than one filesystem.
>
> - Tests that do not have a metadata record are old library tests which
>    which cannot be patched but are printed in a separate table if we
>    request a table to be printed.
>
> - If patching option is selected tests are update with newly calculated
>    timeout. By default we only increase timeouts but that can be
>    overrided with the -o option.
>
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>   scripts/calc_timeouts.py | 133 +++++++++++++++++++++++++++++++++++++++
>   1 file changed, 133 insertions(+)
>   create mode 100755 scripts/calc_timeouts.py
>
> diff --git a/scripts/calc_timeouts.py b/scripts/calc_timeouts.py
> new file mode 100755
> index 000000000..c69ab8f57
> --- /dev/null
> +++ b/scripts/calc_timeouts.py
> @@ -0,0 +1,133 @@
> +#!/usr/bin/python
> +import re
> +import json
> +import getopt
> +import sys
> +from os import system
> +
> +# Top level directory path
> +top_dir = '../'
Better to pass this folder to the script using parameters, also because 
metadata can be found in the install folder as well.
> +
> +# The test runtime is multiplied by this to get a timeout
> +timeout_mul = 1.2
Global variables in python use TIMEOUT_MUL format.
> +
> +def patch(fname, new_timeout, patch_override):
> +
> +    orig_timeout = None
> +    file_path = top_dir + fname
os.path.join()
> +
> +    with open(file_path, 'r') as c_source:
> +        for line in c_source:
> +            timeout = re.search(r'\s*.timeout\s*=\s*(\d+).', line)
This regex should be compiled before loop using re.compile(), otherwise 
re.search() will compile it for each line.
> +            if timeout:
> +                orig_timeout = int(timeout.group(1))
> +
> +    if orig_timeout:
> +        if orig_timeout < new_timeout or patch_override:
> +            print("CHANGE %s timeout %i -> %i" % (fname, orig_timeout, new_timeout))
> +            system("sed -i 's/\\.timeout = [0-9]*/\\.timeout = " + str(new_timeout) + "/' " + file_path)

This can be substituted with a python version of sed, since system() is 
never a good idea, unless we really need to use it for external tools 
which cannot be translated into python. The solution would be:

content = []
matcher = re.match(r'\s*.timeout\s*=\s*(\d+).')
with open(file_path, 'r') as data:
     for line in data:
         # use regex here to find the matching string
         if matcher.search(line):
             content.append(f'timeout = {new_timeout}')
         else:
             content.append(line)

with open(file_path, 'w') as data:
         data.writelines(content)

> +        else:
> +            print("KEEP   %s timeout %i (new %i)" % (fname, orig_timeout, new_timeout))
> +    else:
> +        print("ADD    %s timeout %i" % (fname, new_timeout))
> +        system("sed -i '/static struct tst_test test = {/a\\\\t.timeout = " + str(new_timeout) + ",' " + file_path)
Same here. Maybe we can create a generic sed() function for both cases.
> +
> +def patch_all(timeouts, patch_override):
> +    for timeout in timeouts:
> +        if timeout[3]:
> +            patch(timeout[3], timeout[1], patch_override)
> +
> +def print_table(timeouts):
> +    timeouts.sort(key=lambda x: x[1], reverse=True)
> +
> +    total = 0;
> +
> +    print("Old library tests\n-----------------\n");
> +    for timeout in timeouts:
> +        if not timeout[2]:
> +            print("%-30s %i" % (timeout[0], timeout[1]))
> +            total+=1
> +
> +    print("\n\t%i tests in total" % total)
> +
> +    total = 0;
> +
> +    print("\nNew library tests\n-----------------\n");
> +    for timeout in timeouts:
> +        if timeout[2]:
> +            print("%-30s %i" % (timeout[0], timeout[1]))
> +            total+=1
> +
> +    print("\n\t%i tests in total" % total)
> +
> +def parse_data(results_path):
> +    timeouts = []
> +
> +    with open(results_path, 'r') as file:
> +        results = json.load(file)
> +
> +    with open(top_dir + 'metadata/ltp.json', 'r') as file:
> +        metadata = json.load(file)
> +
> +    for test in results['results']:
> +        name = test['test_fqn']
> +        duration = test['test']['duration']
> +        # If test runs for all_filesystems normalize the runtime per a single filesystem
> +        filesystems = max(1, test['test']['log'].count('TINFO: Formatting /'))
> +        # Check if test is new library test
> +        test_is_newlib = name in metadata['tests']
> +        # Store test file path
> +        path = None if not test_is_newlib else metadata['tests'][name]['fname']
> +        # Filter out tests with runtime
> +        test_has_runtime = False if not test_is_newlib else 'runtime' in metadata['tests'][name]
> +        # Timer tests define runtime dynamically in timer library
> +        if test_is_newlib and 'sample' in metadata['tests'][name]:
> +            test_has_runtime = True
> +        # Select tests that does not have runtime and are executed for longer time
> +        if not test_has_runtime and duration >= 0.5:
> +            timeouts.append((name, int(timeout_mul * duration/filesystems + 0.5), test_is_newlib, path))
> +
> +    return timeouts

In this case it's better to avoid tuples because they make code more 
difficult to read. Hash is a better approach in general:

data["name"] = name
data["timeout"] = int(timeout_mul * duration/filesystems + 0.5)
data["new_lib"] = test_is_newlib
data["path"] = path

return data

> +
> +def print_help():
> +    print('calc_timeouts.py [OPTION] [RESULT].json')
> +    print('\t-h prints this help')
> +    print('\t-o override test timeouts, by default timeouts are only increased')
> +    print('\t-p patch testcases with updated timeouts')
> +    print('\t-t prints table of tests with suggested timeouts')
> +
> +try:
> +    opts, args = getopt.getopt(sys.argv[1:], "hopt")
> +except:
> +    print_help()
> +    sys.exit(1)
> +
> +opt_print_table = False
> +opt_patch_tests = False
> +opt_patch_override = False
> +
> +for opt,arg in opts:
> +    if opt == '-h':
> +        print_help()
> +        sys.exit(0)
> +    if opt == '-o':
> +        opt_patch_override = True
> +    if opt == '-p':
> +        opt_patch_tests = True
> +    if opt == '-t':
> +        opt_print_table = True
> +
> +if not opt_print_table and not opt_patch_tests:
> +    print("No action selected!\n")
> +    print_help()
> +    sys.exit(1)
> +
> +results = args[0] if args else 'results.json'
> +
> +timeouts = parse_data(results)
> +
> +if opt_print_table:
> +    print_table(timeouts)
> +
> +if opt_patch_tests:
> +    patch_all(timeouts, opt_patch_override)
Better to wrap everything into a main() function and to use:

if __name__ == "__main__":

     run()


Andrea


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts
  2025-01-21 13:12   ` Andrea Cervesato via ltp
@ 2025-01-21 14:26     ` Cyril Hrubis
  2025-01-21 14:32       ` Andrea Cervesato via ltp
  0 siblings, 1 reply; 8+ messages in thread
From: Cyril Hrubis @ 2025-01-21 14:26 UTC (permalink / raw)
  To: Andrea Cervesato; +Cc: ltp

Hi!
> I like the general idea, a couple of comments on the python code.

Feel free to push the script with the changes you suggested, you are the
python expert here after all.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts
  2025-01-21 14:26     ` Cyril Hrubis
@ 2025-01-21 14:32       ` Andrea Cervesato via ltp
  0 siblings, 0 replies; 8+ messages in thread
From: Andrea Cervesato via ltp @ 2025-01-21 14:32 UTC (permalink / raw)
  To: Cyril Hrubis; +Cc: ltp

Sure I can do that

On 1/21/25 15:26, Cyril Hrubis wrote:
> Hi!
>> I like the general idea, a couple of comments on the python code.
> Feel free to push the script with the changes you suggested, you are the
> python expert here after all.
>

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [LTP] [PATCH 2/2] syscalls: Update test timeouts
  2025-01-21 12:34 [LTP] [PATCH 0/2] Update test timeouts in an automated way Cyril Hrubis
  2025-01-21 12:34 ` [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts Cyril Hrubis
@ 2025-01-21 12:34 ` Cyril Hrubis
  1 sibling, 0 replies; 8+ messages in thread
From: Cyril Hrubis @ 2025-01-21 12:34 UTC (permalink / raw)
  To: ltp

This change was genered by:

- runnning syscalls via kirk on RPi zero and saving the json result

- running 'cd scripts && calc_timeouts.py -p results.json'

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/kernel/syscalls/access/access01.c                   | 1 +
 testcases/kernel/syscalls/add_key/add_key05.c                 | 1 +
 testcases/kernel/syscalls/alarm/alarm05.c                     | 1 +
 testcases/kernel/syscalls/alarm/alarm06.c                     | 1 +
 testcases/kernel/syscalls/alarm/alarm07.c                     | 1 +
 testcases/kernel/syscalls/bind/bind04.c                       | 1 +
 testcases/kernel/syscalls/bpf/bpf_prog05.c                    | 1 +
 testcases/kernel/syscalls/bpf/bpf_prog06.c                    | 1 +
 testcases/kernel/syscalls/bpf/bpf_prog07.c                    | 1 +
 testcases/kernel/syscalls/cachestat/cachestat01.c             | 1 +
 testcases/kernel/syscalls/cachestat/cachestat04.c             | 1 +
 testcases/kernel/syscalls/chdir/chdir01.c                     | 1 +
 testcases/kernel/syscalls/clock_gettime/leapsec01.c           | 1 +
 testcases/kernel/syscalls/clock_nanosleep/clock_nanosleep01.c | 1 +
 testcases/kernel/syscalls/clock_settime/clock_settime03.c     | 1 +
 testcases/kernel/syscalls/close_range/close_range01.c         | 1 +
 testcases/kernel/syscalls/connect/connect02.c                 | 1 +
 testcases/kernel/syscalls/creat/creat05.c                     | 1 +
 testcases/kernel/syscalls/creat/creat09.c                     | 1 +
 testcases/kernel/syscalls/execve/execve05.c                   | 1 +
 testcases/kernel/syscalls/execveat/execveat03.c               | 1 +
 testcases/kernel/syscalls/fallocate/fallocate04.c             | 1 +
 testcases/kernel/syscalls/fallocate/fallocate05.c             | 1 +
 testcases/kernel/syscalls/fallocate/fallocate06.c             | 2 +-
 testcases/kernel/syscalls/fanotify/fanotify01.c               | 1 +
 testcases/kernel/syscalls/fanotify/fanotify03.c               | 1 +
 testcases/kernel/syscalls/fanotify/fanotify05.c               | 1 +
 testcases/kernel/syscalls/fanotify/fanotify06.c               | 1 +
 testcases/kernel/syscalls/fanotify/fanotify09.c               | 1 +
 testcases/kernel/syscalls/fanotify/fanotify10.c               | 1 +
 testcases/kernel/syscalls/fchmodat2/fchmodat2_01.c            | 1 +
 testcases/kernel/syscalls/fcntl/fcntl14.c                     | 1 +
 testcases/kernel/syscalls/fcntl/fcntl36.c                     | 1 +
 testcases/kernel/syscalls/fdatasync/fdatasync03.c             | 1 +
 testcases/kernel/syscalls/fgetxattr/fgetxattr01.c             | 1 +
 testcases/kernel/syscalls/fremovexattr/fremovexattr01.c       | 1 +
 testcases/kernel/syscalls/fremovexattr/fremovexattr02.c       | 1 +
 testcases/kernel/syscalls/fsconfig/fsconfig01.c               | 1 +
 testcases/kernel/syscalls/fsconfig/fsconfig03.c               | 1 +
 testcases/kernel/syscalls/fsetxattr/fsetxattr01.c             | 1 +
 testcases/kernel/syscalls/fsmount/fsmount01.c                 | 1 +
 testcases/kernel/syscalls/fsmount/fsmount02.c                 | 1 +
 testcases/kernel/syscalls/fsopen/fsopen01.c                   | 1 +
 testcases/kernel/syscalls/fspick/fspick01.c                   | 1 +
 testcases/kernel/syscalls/fspick/fspick02.c                   | 1 +
 testcases/kernel/syscalls/fstatfs/fstatfs01.c                 | 1 +
 testcases/kernel/syscalls/fsync/fsync01.c                     | 1 +
 testcases/kernel/syscalls/fsync/fsync04.c                     | 1 +
 testcases/kernel/syscalls/getpid/getpid01.c                   | 1 +
 testcases/kernel/syscalls/getxattr/getxattr02.c               | 1 +
 testcases/kernel/syscalls/getxattr/getxattr03.c               | 1 +
 testcases/kernel/syscalls/inotify/inotify03.c                 | 1 +
 testcases/kernel/syscalls/inotify/inotify05.c                 | 1 +
 testcases/kernel/syscalls/inotify/inotify07.c                 | 1 +
 testcases/kernel/syscalls/inotify/inotify08.c                 | 1 +
 testcases/kernel/syscalls/inotify/inotify11.c                 | 1 +
 testcases/kernel/syscalls/ioctl/ioctl02.c                     | 1 +
 testcases/kernel/syscalls/ioctl/ioctl04.c                     | 1 +
 testcases/kernel/syscalls/ioctl/ioctl08.c                     | 1 +
 testcases/kernel/syscalls/ioctl/ioctl09.c                     | 1 +
 testcases/kernel/syscalls/ioctl/ioctl_ficlone02.c             | 1 +
 testcases/kernel/syscalls/ioctl/ioctl_ficlone03.c             | 1 +
 testcases/kernel/syscalls/ioctl/ioctl_ficlonerange01.c        | 1 +
 testcases/kernel/syscalls/ioctl/ioctl_ficlonerange02.c        | 1 +
 testcases/kernel/syscalls/ioctl/ioctl_loop01.c                | 1 +
 65 files changed, 65 insertions(+), 1 deletion(-)

diff --git a/testcases/kernel/syscalls/access/access01.c b/testcases/kernel/syscalls/access/access01.c
index 391c8d44b..1c32c6d04 100644
--- a/testcases/kernel/syscalls/access/access01.c
+++ b/testcases/kernel/syscalls/access/access01.c
@@ -314,6 +314,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.needs_tmpdir = 1,
 	.needs_root = 1,
 	.forks_child = 1,
diff --git a/testcases/kernel/syscalls/add_key/add_key05.c b/testcases/kernel/syscalls/add_key/add_key05.c
index 3abd58b83..c9a2f840e 100644
--- a/testcases/kernel/syscalls/add_key/add_key05.c
+++ b/testcases/kernel/syscalls/add_key/add_key05.c
@@ -212,6 +212,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 2,
 	.test = do_test,
 	.tcnt = 2,
 	.needs_root = 1,
diff --git a/testcases/kernel/syscalls/alarm/alarm05.c b/testcases/kernel/syscalls/alarm/alarm05.c
index 2eeb1c22f..82bff69cb 100644
--- a/testcases/kernel/syscalls/alarm/alarm05.c
+++ b/testcases/kernel/syscalls/alarm/alarm05.c
@@ -44,6 +44,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 2,
 	.test_all = run,
 	.setup = setup,
 };
diff --git a/testcases/kernel/syscalls/alarm/alarm06.c b/testcases/kernel/syscalls/alarm/alarm06.c
index 82c0d44bd..2cfa26207 100644
--- a/testcases/kernel/syscalls/alarm/alarm06.c
+++ b/testcases/kernel/syscalls/alarm/alarm06.c
@@ -41,6 +41,7 @@ static void verify_alarm(void)
 }
 
 static struct tst_test test = {
+	.timeout = 4,
 	.setup = setup,
 	.test_all = verify_alarm,
 };
diff --git a/testcases/kernel/syscalls/alarm/alarm07.c b/testcases/kernel/syscalls/alarm/alarm07.c
index 64aed507d..c04745f9d 100644
--- a/testcases/kernel/syscalls/alarm/alarm07.c
+++ b/testcases/kernel/syscalls/alarm/alarm07.c
@@ -47,6 +47,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 4,
 	.test_all = verify_alarm,
 	.setup = setup,
 	.forks_child = 1,
diff --git a/testcases/kernel/syscalls/bind/bind04.c b/testcases/kernel/syscalls/bind/bind04.c
index d8456e739..2a46559de 100644
--- a/testcases/kernel/syscalls/bind/bind04.c
+++ b/testcases/kernel/syscalls/bind/bind04.c
@@ -161,6 +161,7 @@ static void test_bind(unsigned int n)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test = test_bind,
 	.tcnt = ARRAY_SIZE(testcase_list),
 	.needs_tmpdir = 1,
diff --git a/testcases/kernel/syscalls/bpf/bpf_prog05.c b/testcases/kernel/syscalls/bpf/bpf_prog05.c
index 08254ba89..050081712 100644
--- a/testcases/kernel/syscalls/bpf/bpf_prog05.c
+++ b/testcases/kernel/syscalls/bpf/bpf_prog05.c
@@ -190,6 +190,7 @@ static void run(void)
 }
 
 static struct tst_test test = {
+	.timeout = 20,
 	.setup = setup,
 	.test_all = run,
 	.taint_check = TST_TAINT_W | TST_TAINT_D,
diff --git a/testcases/kernel/syscalls/bpf/bpf_prog06.c b/testcases/kernel/syscalls/bpf/bpf_prog06.c
index cee9616cf..87fdd8060 100644
--- a/testcases/kernel/syscalls/bpf/bpf_prog06.c
+++ b/testcases/kernel/syscalls/bpf/bpf_prog06.c
@@ -131,6 +131,7 @@ static void run(void)
 }
 
 static struct tst_test test = {
+	.timeout = 20,
 	.setup = setup,
 	.test_all = run,
 	.min_kver = "5.8",
diff --git a/testcases/kernel/syscalls/bpf/bpf_prog07.c b/testcases/kernel/syscalls/bpf/bpf_prog07.c
index dab5bb8ad..29dfbf2ca 100644
--- a/testcases/kernel/syscalls/bpf/bpf_prog07.c
+++ b/testcases/kernel/syscalls/bpf/bpf_prog07.c
@@ -139,6 +139,7 @@ static void run(void)
 }
 
 static struct tst_test test = {
+	.timeout = 20,
 	.setup = setup,
 	.test_all = run,
 	.min_kver = "5.8",
diff --git a/testcases/kernel/syscalls/cachestat/cachestat01.c b/testcases/kernel/syscalls/cachestat/cachestat01.c
index 2dca68885..22f0071d1 100644
--- a/testcases/kernel/syscalls/cachestat/cachestat01.c
+++ b/testcases/kernel/syscalls/cachestat/cachestat01.c
@@ -85,6 +85,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 13,
 	.test = run,
 	.tcnt = 2,
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/cachestat/cachestat04.c b/testcases/kernel/syscalls/cachestat/cachestat04.c
index a59494451..7cef9897d 100644
--- a/testcases/kernel/syscalls/cachestat/cachestat04.c
+++ b/testcases/kernel/syscalls/cachestat/cachestat04.c
@@ -47,6 +47,7 @@ static void run(void)
 }
 
 static struct tst_test test = {
+	.timeout = 2,
 	.test_all = run,
 	.mount_device = 1,
 	.mntpoint = MNTPOINT,
diff --git a/testcases/kernel/syscalls/chdir/chdir01.c b/testcases/kernel/syscalls/chdir/chdir01.c
index d50a8f50c..1c1299332 100644
--- a/testcases/kernel/syscalls/chdir/chdir01.c
+++ b/testcases/kernel/syscalls/chdir/chdir01.c
@@ -149,6 +149,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.needs_root = 1,
 	.mount_device = 1,
 	.mntpoint = MNTPOINT,
diff --git a/testcases/kernel/syscalls/clock_gettime/leapsec01.c b/testcases/kernel/syscalls/clock_gettime/leapsec01.c
index e623b4107..4cd469dc2 100644
--- a/testcases/kernel/syscalls/clock_gettime/leapsec01.c
+++ b/testcases/kernel/syscalls/clock_gettime/leapsec01.c
@@ -196,6 +196,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 40,
 	.test_all = run_leapsec,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/clock_nanosleep/clock_nanosleep01.c b/testcases/kernel/syscalls/clock_nanosleep/clock_nanosleep01.c
index eef8a5992..55e7b294e 100644
--- a/testcases/kernel/syscalls/clock_nanosleep/clock_nanosleep01.c
+++ b/testcases/kernel/syscalls/clock_nanosleep/clock_nanosleep01.c
@@ -219,6 +219,7 @@ static void do_test(unsigned int i)
 }
 
 static struct tst_test test = {
+	.timeout = 3,
 	.tcnt = ARRAY_SIZE(tcase),
 	.test = do_test,
 	.test_variants = ARRAY_SIZE(variants),
diff --git a/testcases/kernel/syscalls/clock_settime/clock_settime03.c b/testcases/kernel/syscalls/clock_settime/clock_settime03.c
index f196a257c..692e63788 100644
--- a/testcases/kernel/syscalls/clock_settime/clock_settime03.c
+++ b/testcases/kernel/syscalls/clock_settime/clock_settime03.c
@@ -104,6 +104,7 @@ static void run(void)
 }
 
 static struct tst_test test = {
+	.timeout = 4,
 	.test_all = run,
 	.test_variants = ARRAY_SIZE(variants),
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/close_range/close_range01.c b/testcases/kernel/syscalls/close_range/close_range01.c
index 110ffa9b7..941f3e403 100644
--- a/testcases/kernel/syscalls/close_range/close_range01.c
+++ b/testcases/kernel/syscalls/close_range/close_range01.c
@@ -191,6 +191,7 @@ static void run(unsigned int n)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.tcnt = 4,
 	.forks_child = 1,
 	.mount_device = 1,
diff --git a/testcases/kernel/syscalls/connect/connect02.c b/testcases/kernel/syscalls/connect/connect02.c
index e20214e24..087b514bf 100644
--- a/testcases/kernel/syscalls/connect/connect02.c
+++ b/testcases/kernel/syscalls/connect/connect02.c
@@ -126,6 +126,7 @@ static void run(void)
 }
 
 static struct tst_test test = {
+	.timeout = 3,
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/creat/creat05.c b/testcases/kernel/syscalls/creat/creat05.c
index bf4099439..32074a449 100644
--- a/testcases/kernel/syscalls/creat/creat05.c
+++ b/testcases/kernel/syscalls/creat/creat05.c
@@ -74,6 +74,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test_all = verify_creat,
 	.needs_tmpdir = 1,
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/creat/creat09.c b/testcases/kernel/syscalls/creat/creat09.c
index a5d3740ac..d18165772 100644
--- a/testcases/kernel/syscalls/creat/creat09.c
+++ b/testcases/kernel/syscalls/creat/creat09.c
@@ -138,6 +138,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test = run,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/execve/execve05.c b/testcases/kernel/syscalls/execve/execve05.c
index d87d7446d..530c7cc89 100644
--- a/testcases/kernel/syscalls/execve/execve05.c
+++ b/testcases/kernel/syscalls/execve/execve05.c
@@ -62,6 +62,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 3,
 	.test_all = verify_execve,
 	.options = (struct tst_option[]) {
 		{"n:", &opt_nchild, "Numbers of children"},
diff --git a/testcases/kernel/syscalls/execveat/execveat03.c b/testcases/kernel/syscalls/execveat/execveat03.c
index 057d83278..684f0d0c6 100644
--- a/testcases/kernel/syscalls/execveat/execveat03.c
+++ b/testcases/kernel/syscalls/execveat/execveat03.c
@@ -68,6 +68,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.needs_root = 1,
 	.mount_device = 1,
 	.needs_overlay = 1,
diff --git a/testcases/kernel/syscalls/fallocate/fallocate04.c b/testcases/kernel/syscalls/fallocate/fallocate04.c
index 7b285fb5d..3a8ea5fa7 100644
--- a/testcases/kernel/syscalls/fallocate/fallocate04.c
+++ b/testcases/kernel/syscalls/fallocate/fallocate04.c
@@ -282,6 +282,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.options = (struct tst_option[]) {
 		{"v", &verbose, "Turns on verbose mode"},
 		{}
diff --git a/testcases/kernel/syscalls/fallocate/fallocate05.c b/testcases/kernel/syscalls/fallocate/fallocate05.c
index 732a2f15d..f17cc993e 100644
--- a/testcases/kernel/syscalls/fallocate/fallocate05.c
+++ b/testcases/kernel/syscalls/fallocate/fallocate05.c
@@ -172,6 +172,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 42,
 	.needs_root = 1,
 	.mount_device = 1,
 	.mntpoint = MNTPOINT,
diff --git a/testcases/kernel/syscalls/fallocate/fallocate06.c b/testcases/kernel/syscalls/fallocate/fallocate06.c
index 5e1454c30..0e1509ace 100644
--- a/testcases/kernel/syscalls/fallocate/fallocate06.c
+++ b/testcases/kernel/syscalls/fallocate/fallocate06.c
@@ -261,7 +261,7 @@ static struct tst_test test = {
 	.tcnt = ARRAY_SIZE(testcase_list),
 	.needs_root = 1,
 	.dev_min_size = 1024,
-	.timeout = 120,
+	.timeout = 150,
 	.mount_device = 1,
 	.mntpoint = MNTPOINT,
 	.all_filesystems = 1,
diff --git a/testcases/kernel/syscalls/fanotify/fanotify01.c b/testcases/kernel/syscalls/fanotify/fanotify01.c
index 5d18fe851..c7e759166 100644
--- a/testcases/kernel/syscalls/fanotify/fanotify01.c
+++ b/testcases/kernel/syscalls/fanotify/fanotify01.c
@@ -387,6 +387,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.test = test_fanotify,
 	.tcnt = ARRAY_SIZE(tcases),
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/fanotify/fanotify03.c b/testcases/kernel/syscalls/fanotify/fanotify03.c
index 3ed78e5fa..64c933c19 100644
--- a/testcases/kernel/syscalls/fanotify/fanotify03.c
+++ b/testcases/kernel/syscalls/fanotify/fanotify03.c
@@ -343,6 +343,7 @@ static const char *const resource_files[] = {
 };
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test = test_fanotify,
 	.tcnt = ARRAY_SIZE(tcases),
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/fanotify/fanotify05.c b/testcases/kernel/syscalls/fanotify/fanotify05.c
index 12c240881..435a91c49 100644
--- a/testcases/kernel/syscalls/fanotify/fanotify05.c
+++ b/testcases/kernel/syscalls/fanotify/fanotify05.c
@@ -208,6 +208,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 13,
 	.test = test_fanotify,
 	.tcnt = ARRAY_SIZE(tcases),
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/fanotify/fanotify06.c b/testcases/kernel/syscalls/fanotify/fanotify06.c
index 8779e34d9..b4992afa0 100644
--- a/testcases/kernel/syscalls/fanotify/fanotify06.c
+++ b/testcases/kernel/syscalls/fanotify/fanotify06.c
@@ -236,6 +236,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test = test_fanotify,
 	.tcnt = ARRAY_SIZE(tcases),
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/fanotify/fanotify09.c b/testcases/kernel/syscalls/fanotify/fanotify09.c
index 48b198b94..45303c31e 100644
--- a/testcases/kernel/syscalls/fanotify/fanotify09.c
+++ b/testcases/kernel/syscalls/fanotify/fanotify09.c
@@ -507,6 +507,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test = test_fanotify,
 	.tcnt = ARRAY_SIZE(tcases),
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/fanotify/fanotify10.c b/testcases/kernel/syscalls/fanotify/fanotify10.c
index eedd1442f..23e9554b3 100644
--- a/testcases/kernel/syscalls/fanotify/fanotify10.c
+++ b/testcases/kernel/syscalls/fanotify/fanotify10.c
@@ -959,6 +959,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.test = test_fanotify,
 	.tcnt = ARRAY_SIZE(tcases),
 	.test_variants = 2,
diff --git a/testcases/kernel/syscalls/fchmodat2/fchmodat2_01.c b/testcases/kernel/syscalls/fchmodat2/fchmodat2_01.c
index f026b18df..e58be31c4 100644
--- a/testcases/kernel/syscalls/fchmodat2/fchmodat2_01.c
+++ b/testcases/kernel/syscalls/fchmodat2/fchmodat2_01.c
@@ -104,6 +104,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/fcntl/fcntl14.c b/testcases/kernel/syscalls/fcntl/fcntl14.c
index 367701d68..21dbede5c 100644
--- a/testcases/kernel/syscalls/fcntl/fcntl14.c
+++ b/testcases/kernel/syscalls/fcntl/fcntl14.c
@@ -263,6 +263,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 8,
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/fcntl/fcntl36.c b/testcases/kernel/syscalls/fcntl/fcntl36.c
index fabdbe905..19a5a0303 100644
--- a/testcases/kernel/syscalls/fcntl/fcntl36.c
+++ b/testcases/kernel/syscalls/fcntl/fcntl36.c
@@ -387,6 +387,7 @@ static void tests(unsigned int i)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.needs_tmpdir = 1,
 	.test = tests,
 	.tcnt = ARRAY_SIZE(tcases),
diff --git a/testcases/kernel/syscalls/fdatasync/fdatasync03.c b/testcases/kernel/syscalls/fdatasync/fdatasync03.c
index 5f3e0c96f..f1b5f0fb8 100644
--- a/testcases/kernel/syscalls/fdatasync/fdatasync03.c
+++ b/testcases/kernel/syscalls/fdatasync/fdatasync03.c
@@ -53,6 +53,7 @@ static void verify_fdatasync(void)
 }
 
 static struct tst_test test = {
+	.timeout = 15,
 	.needs_root = 1,
 	.mount_device = 1,
 	.all_filesystems = 1,
diff --git a/testcases/kernel/syscalls/fgetxattr/fgetxattr01.c b/testcases/kernel/syscalls/fgetxattr/fgetxattr01.c
index 52e6e44ab..eefddd829 100644
--- a/testcases/kernel/syscalls/fgetxattr/fgetxattr01.c
+++ b/testcases/kernel/syscalls/fgetxattr/fgetxattr01.c
@@ -134,6 +134,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.setup = setup,
 	.test = verify_fgetxattr,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/fremovexattr/fremovexattr01.c b/testcases/kernel/syscalls/fremovexattr/fremovexattr01.c
index 907d210d1..aca2fce7c 100644
--- a/testcases/kernel/syscalls/fremovexattr/fremovexattr01.c
+++ b/testcases/kernel/syscalls/fremovexattr/fremovexattr01.c
@@ -83,6 +83,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 12,
 	.setup = setup,
 	.test_all = verify_fremovexattr,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/fremovexattr/fremovexattr02.c b/testcases/kernel/syscalls/fremovexattr/fremovexattr02.c
index eb106a8fa..78af8be32 100644
--- a/testcases/kernel/syscalls/fremovexattr/fremovexattr02.c
+++ b/testcases/kernel/syscalls/fremovexattr/fremovexattr02.c
@@ -105,6 +105,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.setup = setup,
 	.test = verify_fremovexattr,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/fsconfig/fsconfig01.c b/testcases/kernel/syscalls/fsconfig/fsconfig01.c
index a585daa6d..678d21815 100644
--- a/testcases/kernel/syscalls/fsconfig/fsconfig01.c
+++ b/testcases/kernel/syscalls/fsconfig/fsconfig01.c
@@ -82,6 +82,7 @@ static void run(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.test_all = run,
 	.setup = fsopen_supported_by_kernel,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/fsconfig/fsconfig03.c b/testcases/kernel/syscalls/fsconfig/fsconfig03.c
index 0ba5355d3..503753636 100644
--- a/testcases/kernel/syscalls/fsconfig/fsconfig03.c
+++ b/testcases/kernel/syscalls/fsconfig/fsconfig03.c
@@ -80,6 +80,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/fsetxattr/fsetxattr01.c b/testcases/kernel/syscalls/fsetxattr/fsetxattr01.c
index b65b27bdf..73e1fcfb7 100644
--- a/testcases/kernel/syscalls/fsetxattr/fsetxattr01.c
+++ b/testcases/kernel/syscalls/fsetxattr/fsetxattr01.c
@@ -214,6 +214,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.setup = setup,
 	.test = verify_fsetxattr,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/fsmount/fsmount01.c b/testcases/kernel/syscalls/fsmount/fsmount01.c
index 5f755863f..6dafe25a8 100644
--- a/testcases/kernel/syscalls/fsmount/fsmount01.c
+++ b/testcases/kernel/syscalls/fsmount/fsmount01.c
@@ -88,6 +88,7 @@ static void run(unsigned int n)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.tcnt = ARRAY_SIZE(tcases),
 	.test = run,
 	.setup = fsopen_supported_by_kernel,
diff --git a/testcases/kernel/syscalls/fsmount/fsmount02.c b/testcases/kernel/syscalls/fsmount/fsmount02.c
index a4f42dc18..55f0e2f28 100644
--- a/testcases/kernel/syscalls/fsmount/fsmount02.c
+++ b/testcases/kernel/syscalls/fsmount/fsmount02.c
@@ -68,6 +68,7 @@ static void run(unsigned int n)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.tcnt = ARRAY_SIZE(tcases),
 	.test = run,
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/fsopen/fsopen01.c b/testcases/kernel/syscalls/fsopen/fsopen01.c
index c2c719c96..9dd87b99f 100644
--- a/testcases/kernel/syscalls/fsopen/fsopen01.c
+++ b/testcases/kernel/syscalls/fsopen/fsopen01.c
@@ -69,6 +69,7 @@ out:
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.tcnt = ARRAY_SIZE(tcases),
 	.test = run,
 	.setup = fsopen_supported_by_kernel,
diff --git a/testcases/kernel/syscalls/fspick/fspick01.c b/testcases/kernel/syscalls/fspick/fspick01.c
index d3309a912..d03cacd3d 100644
--- a/testcases/kernel/syscalls/fspick/fspick01.c
+++ b/testcases/kernel/syscalls/fspick/fspick01.c
@@ -56,6 +56,7 @@ out:
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.tcnt = ARRAY_SIZE(tcases),
 	.test = run,
 	.setup = fsopen_supported_by_kernel,
diff --git a/testcases/kernel/syscalls/fspick/fspick02.c b/testcases/kernel/syscalls/fspick/fspick02.c
index f9a3697c1..89bdd2cc8 100644
--- a/testcases/kernel/syscalls/fspick/fspick02.c
+++ b/testcases/kernel/syscalls/fspick/fspick02.c
@@ -43,6 +43,7 @@ static void run(unsigned int n)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.tcnt = ARRAY_SIZE(tcases),
 	.test = run,
 	.setup = fsopen_supported_by_kernel,
diff --git a/testcases/kernel/syscalls/fstatfs/fstatfs01.c b/testcases/kernel/syscalls/fstatfs/fstatfs01.c
index 9d3909ace..044ab6548 100644
--- a/testcases/kernel/syscalls/fstatfs/fstatfs01.c
+++ b/testcases/kernel/syscalls/fstatfs/fstatfs01.c
@@ -56,6 +56,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.setup = setup,
 	.cleanup = cleanup,
 	.tcnt = ARRAY_SIZE(tcases),
diff --git a/testcases/kernel/syscalls/fsync/fsync01.c b/testcases/kernel/syscalls/fsync/fsync01.c
index 072245fc8..7ae5296c7 100644
--- a/testcases/kernel/syscalls/fsync/fsync01.c
+++ b/testcases/kernel/syscalls/fsync/fsync01.c
@@ -44,6 +44,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.cleanup = cleanup,
 	.setup = setup,
 	.test_all = verify_fsync,
diff --git a/testcases/kernel/syscalls/fsync/fsync04.c b/testcases/kernel/syscalls/fsync/fsync04.c
index 9aa1584c1..f7553ff58 100644
--- a/testcases/kernel/syscalls/fsync/fsync04.c
+++ b/testcases/kernel/syscalls/fsync/fsync04.c
@@ -53,6 +53,7 @@ static void verify_fsync(void)
 }
 
 static struct tst_test test = {
+	.timeout = 17,
 	.needs_root = 1,
 	.mount_device = 1,
 	.all_filesystems = 1,
diff --git a/testcases/kernel/syscalls/getpid/getpid01.c b/testcases/kernel/syscalls/getpid/getpid01.c
index 495002037..314c45d09 100644
--- a/testcases/kernel/syscalls/getpid/getpid01.c
+++ b/testcases/kernel/syscalls/getpid/getpid01.c
@@ -43,6 +43,7 @@ static void verify_getpid(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.setup = setup,
 	.forks_child = 1,
 	.test_all = verify_getpid,
diff --git a/testcases/kernel/syscalls/getxattr/getxattr02.c b/testcases/kernel/syscalls/getxattr/getxattr02.c
index 5a84d876c..593ad7b8c 100644
--- a/testcases/kernel/syscalls/getxattr/getxattr02.c
+++ b/testcases/kernel/syscalls/getxattr/getxattr02.c
@@ -105,6 +105,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.all_filesystems = 1,
 	.needs_root = 1,
 	.mntpoint = MNTPOINT,
diff --git a/testcases/kernel/syscalls/getxattr/getxattr03.c b/testcases/kernel/syscalls/getxattr/getxattr03.c
index 1fe4ba48f..85f26402a 100644
--- a/testcases/kernel/syscalls/getxattr/getxattr03.c
+++ b/testcases/kernel/syscalls/getxattr/getxattr03.c
@@ -37,6 +37,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 14,
 	.all_filesystems = 1,
 	.needs_root = 1,
 	.mntpoint = MNTPOINT,
diff --git a/testcases/kernel/syscalls/inotify/inotify03.c b/testcases/kernel/syscalls/inotify/inotify03.c
index ff025b360..4ee5e9461 100644
--- a/testcases/kernel/syscalls/inotify/inotify03.c
+++ b/testcases/kernel/syscalls/inotify/inotify03.c
@@ -166,6 +166,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.needs_root = 1,
 	.format_device = 1,
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/inotify/inotify05.c b/testcases/kernel/syscalls/inotify/inotify05.c
index a1597217f..38b30878c 100644
--- a/testcases/kernel/syscalls/inotify/inotify05.c
+++ b/testcases/kernel/syscalls/inotify/inotify05.c
@@ -148,6 +148,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.needs_tmpdir = 1,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/inotify/inotify07.c b/testcases/kernel/syscalls/inotify/inotify07.c
index 08ea1e06a..bcfc569a8 100644
--- a/testcases/kernel/syscalls/inotify/inotify07.c
+++ b/testcases/kernel/syscalls/inotify/inotify07.c
@@ -182,6 +182,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.needs_root = 1,
 	.mount_device = 1,
 	.needs_overlay = 1,
diff --git a/testcases/kernel/syscalls/inotify/inotify08.c b/testcases/kernel/syscalls/inotify/inotify08.c
index 9c2ecdabd..149a0a1f8 100644
--- a/testcases/kernel/syscalls/inotify/inotify08.c
+++ b/testcases/kernel/syscalls/inotify/inotify08.c
@@ -176,6 +176,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.needs_root = 1,
 	.mount_device = 1,
 	.needs_overlay = 1,
diff --git a/testcases/kernel/syscalls/inotify/inotify11.c b/testcases/kernel/syscalls/inotify/inotify11.c
index 1630477fc..2e571f3a0 100644
--- a/testcases/kernel/syscalls/inotify/inotify11.c
+++ b/testcases/kernel/syscalls/inotify/inotify11.c
@@ -119,6 +119,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 12,
 	.needs_tmpdir = 1,
 	.forks_child = 1,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/ioctl/ioctl02.c b/testcases/kernel/syscalls/ioctl/ioctl02.c
index f3bfb239a..9337da384 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl02.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl02.c
@@ -242,6 +242,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 9,
 	.needs_root = 1,
 	.needs_checkpoints = 1,
 	.forks_child = 1,
diff --git a/testcases/kernel/syscalls/ioctl/ioctl04.c b/testcases/kernel/syscalls/ioctl/ioctl04.c
index 262c06e97..f96b22f0c 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl04.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl04.c
@@ -84,6 +84,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.format_device = 1,
 	.needs_root = 1,
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/ioctl/ioctl08.c b/testcases/kernel/syscalls/ioctl/ioctl08.c
index 834c5ae74..14f9354c9 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl08.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl08.c
@@ -118,6 +118,7 @@ static void setup(void)
 
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test = verify_ioctl,
 	.tcnt = ARRAY_SIZE(tcases),
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/ioctl/ioctl09.c b/testcases/kernel/syscalls/ioctl/ioctl09.c
index 9c7921086..0d1f1072d 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl09.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl09.c
@@ -110,6 +110,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_ioctl,
diff --git a/testcases/kernel/syscalls/ioctl/ioctl_ficlone02.c b/testcases/kernel/syscalls/ioctl/ioctl_ficlone02.c
index fab0daaee..e5d4be96f 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl_ficlone02.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl_ficlone02.c
@@ -51,6 +51,7 @@ static void setup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 10,
 	.test_all = run,
 	.setup = setup,
 	.min_kver = "4.5",
diff --git a/testcases/kernel/syscalls/ioctl/ioctl_ficlone03.c b/testcases/kernel/syscalls/ioctl/ioctl_ficlone03.c
index f12c4e7fd..b0a0f8d63 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl_ficlone03.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl_ficlone03.c
@@ -101,6 +101,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test = run,
 	.tcnt = ARRAY_SIZE(tcases),
 	.setup = setup,
diff --git a/testcases/kernel/syscalls/ioctl/ioctl_ficlonerange01.c b/testcases/kernel/syscalls/ioctl/ioctl_ficlonerange01.c
index 8fd1d299a..707270765 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl_ficlonerange01.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl_ficlonerange01.c
@@ -137,6 +137,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/ioctl/ioctl_ficlonerange02.c b/testcases/kernel/syscalls/ioctl/ioctl_ficlonerange02.c
index eb941c3fc..cc1d0e031 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl_ficlonerange02.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl_ficlonerange02.c
@@ -74,6 +74,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 4,
 	.test_all = run,
 	.setup = setup,
 	.cleanup = cleanup,
diff --git a/testcases/kernel/syscalls/ioctl/ioctl_loop01.c b/testcases/kernel/syscalls/ioctl/ioctl_loop01.c
index e7b337e4a..c3fb9f583 100644
--- a/testcases/kernel/syscalls/ioctl/ioctl_loop01.c
+++ b/testcases/kernel/syscalls/ioctl/ioctl_loop01.c
@@ -140,6 +140,7 @@ static void cleanup(void)
 }
 
 static struct tst_test test = {
+	.timeout = 1,
 	.setup = setup,
 	.cleanup = cleanup,
 	.test_all = verify_ioctl_loop,
-- 
2.45.2


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [LTP] [PATCH 0/2] Update test timeouts in an automated way
@ 2025-01-22 12:48 Andrea Cervesato
  2025-01-22 12:48 ` [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts Andrea Cervesato
  0 siblings, 1 reply; 8+ messages in thread
From: Andrea Cervesato @ 2025-01-22 12:48 UTC (permalink / raw)
  To: ltp

First patch adds a script that can parse kirk test results, calculate
timeouts and patch tests with the newly calculated timeouts.

Second patch updates tests timeouts using this script with a results
captured on RPi zero.

Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
Cyril Hrubis (2):
      scripts: Add simple script for calculating timeouts
      syscalls: Update test timeouts

 scripts/calctimeouts.py                            | 232 +++++++++++++++++++++
 testcases/kernel/syscalls/access/access01.c        |   1 +
 testcases/kernel/syscalls/add_key/add_key05.c      |   1 +
 testcases/kernel/syscalls/alarm/alarm05.c          |   1 +
 testcases/kernel/syscalls/alarm/alarm06.c          |   1 +
 testcases/kernel/syscalls/alarm/alarm07.c          |   1 +
 testcases/kernel/syscalls/bind/bind04.c            |   1 +
 testcases/kernel/syscalls/bpf/bpf_prog05.c         |   1 +
 testcases/kernel/syscalls/bpf/bpf_prog06.c         |   1 +
 testcases/kernel/syscalls/bpf/bpf_prog07.c         |   1 +
 testcases/kernel/syscalls/cachestat/cachestat01.c  |   1 +
 testcases/kernel/syscalls/cachestat/cachestat04.c  |   1 +
 testcases/kernel/syscalls/chdir/chdir01.c          |   1 +
 .../kernel/syscalls/clock_gettime/leapsec01.c      |   1 +
 .../syscalls/clock_nanosleep/clock_nanosleep01.c   |   1 +
 .../syscalls/clock_settime/clock_settime03.c       |   1 +
 .../kernel/syscalls/close_range/close_range01.c    |   1 +
 testcases/kernel/syscalls/connect/connect02.c      |   1 +
 testcases/kernel/syscalls/creat/creat05.c          |   1 +
 testcases/kernel/syscalls/creat/creat09.c          |   1 +
 testcases/kernel/syscalls/execve/execve05.c        |   1 +
 testcases/kernel/syscalls/execveat/execveat03.c    |   1 +
 testcases/kernel/syscalls/fallocate/fallocate04.c  |   1 +
 testcases/kernel/syscalls/fallocate/fallocate05.c  |   1 +
 testcases/kernel/syscalls/fallocate/fallocate06.c  |   2 +-
 testcases/kernel/syscalls/fanotify/fanotify01.c    |   1 +
 testcases/kernel/syscalls/fanotify/fanotify03.c    |   1 +
 testcases/kernel/syscalls/fanotify/fanotify05.c    |   1 +
 testcases/kernel/syscalls/fanotify/fanotify06.c    |   1 +
 testcases/kernel/syscalls/fanotify/fanotify09.c    |   1 +
 testcases/kernel/syscalls/fanotify/fanotify10.c    |   1 +
 testcases/kernel/syscalls/fchmodat2/fchmodat2_01.c |   1 +
 testcases/kernel/syscalls/fcntl/fcntl14.c          |   1 +
 testcases/kernel/syscalls/fcntl/fcntl36.c          |   1 +
 testcases/kernel/syscalls/fdatasync/fdatasync03.c  |   1 +
 testcases/kernel/syscalls/fgetxattr/fgetxattr01.c  |   1 +
 .../kernel/syscalls/fremovexattr/fremovexattr01.c  |   1 +
 .../kernel/syscalls/fremovexattr/fremovexattr02.c  |   1 +
 testcases/kernel/syscalls/fsconfig/fsconfig01.c    |   1 +
 testcases/kernel/syscalls/fsconfig/fsconfig03.c    |   1 +
 testcases/kernel/syscalls/fsetxattr/fsetxattr01.c  |   1 +
 testcases/kernel/syscalls/fsmount/fsmount01.c      |   1 +
 testcases/kernel/syscalls/fsmount/fsmount02.c      |   1 +
 testcases/kernel/syscalls/fsopen/fsopen01.c        |   1 +
 testcases/kernel/syscalls/fspick/fspick01.c        |   1 +
 testcases/kernel/syscalls/fspick/fspick02.c        |   1 +
 testcases/kernel/syscalls/fstatfs/fstatfs01.c      |   1 +
 testcases/kernel/syscalls/fsync/fsync01.c          |   1 +
 testcases/kernel/syscalls/fsync/fsync04.c          |   1 +
 testcases/kernel/syscalls/getpid/getpid01.c        |   1 +
 testcases/kernel/syscalls/getxattr/getxattr02.c    |   1 +
 testcases/kernel/syscalls/getxattr/getxattr03.c    |   1 +
 testcases/kernel/syscalls/inotify/inotify03.c      |   1 +
 testcases/kernel/syscalls/inotify/inotify05.c      |   1 +
 testcases/kernel/syscalls/inotify/inotify07.c      |   1 +
 testcases/kernel/syscalls/inotify/inotify08.c      |   1 +
 testcases/kernel/syscalls/inotify/inotify11.c      |   1 +
 testcases/kernel/syscalls/ioctl/ioctl02.c          |   1 +
 testcases/kernel/syscalls/ioctl/ioctl04.c          |   1 +
 testcases/kernel/syscalls/ioctl/ioctl08.c          |   1 +
 testcases/kernel/syscalls/ioctl/ioctl09.c          |   1 +
 testcases/kernel/syscalls/ioctl/ioctl_ficlone02.c  |   1 +
 testcases/kernel/syscalls/ioctl/ioctl_ficlone03.c  |   1 +
 .../kernel/syscalls/ioctl/ioctl_ficlonerange01.c   |   1 +
 .../kernel/syscalls/ioctl/ioctl_ficlonerange02.c   |   1 +
 testcases/kernel/syscalls/ioctl/ioctl_loop01.c     |   1 +
 66 files changed, 297 insertions(+), 1 deletion(-)
---
base-commit: d3df587ed1ee3f881f802f5060506a8192c38e0d
change-id: 20250121-cyril_script_update_timeouts-4acc3c00b3dc

Best regards,
-- 
Andrea Cervesato <andrea.cervesato@suse.com>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts
  2025-01-22 12:48 [LTP] [PATCH 0/2] Update test timeouts in an automated way Andrea Cervesato
@ 2025-01-22 12:48 ` Andrea Cervesato
  2025-01-22 13:13   ` Cyril Hrubis
  0 siblings, 1 reply; 8+ messages in thread
From: Andrea Cervesato @ 2025-01-22 12:48 UTC (permalink / raw)
  To: ltp

From: Cyril Hrubis <chrubis@suse.cz>

This script parses JSON results from kirk and LTP metadata in order
calculate timeouts for tests based on the result file. It can also patch
tests automatically.

The script does:

- Take the results and pick all tests that run for longer than 0.5s.
  Multiplie the time with a constant (currently 1.2) to get a suggested
  timeout.

- Exclude tests that have runtime defined since these are controlled
  by the runtime (that filters out all fuzzy sync tests).

  There is a special case for timer tests that defines runtime only
  dynamically in the timer library code. This should be possibly fixed
  with special value for the .runtime in tst_test. E.g.
  TST_RUNTIME_DYNAMIC for tests that only set runtime in the setup.

- Normalize the timeout for a single filesystem run if test is running for
  more than one filesystem.

- Verify if tests are build on top of old library by checking at
  metadata file

- Update test with a  with newly calculated timeout.
  By default we only increase timeouts but that can be overridden using
  the -o option.

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
Co-developed-by: Andrea Cervesato <andrea.cervesato@suse.com>
Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
 scripts/calctimeouts.py | 232 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 232 insertions(+)

diff --git a/scripts/calctimeouts.py b/scripts/calctimeouts.py
new file mode 100755
index 0000000000000000000000000000000000000000..d5e8fe2c15faad7c8d1ec8c15541f35a97a8c0c4
--- /dev/null
+++ b/scripts/calctimeouts.py
@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+"""
+This script parses JSON results from kirk and LTP metadata in order to
+calculate timeouts for tests based on the results file.
+It can also patch tests automatically and replace the calculated timeout.
+"""
+
+import re
+import os
+import json
+import argparse
+
+# The test runtime is multiplied by this to get a timeout
+TIMEOUT_MUL = 1.2
+
+
+def _sed(fname, expr, replace):
+    """
+    Pythonic version of sed command.
+    """
+    content = []
+    matcher = re.compile(expr)
+
+    with open(fname, 'r', encoding="utf-8") as data:
+        for line in data:
+            match = matcher.search(line)
+            if not match:
+                content.append(line)
+            else:
+                content.append(replace)
+
+    with open(fname, 'w', encoding="utf-8") as data:
+        data.writelines(content)
+
+
+def _patch(ltp_dir, fname, new_timeout, override):
+    """
+    If `override` is True, it patches a test file, searching for timeout and
+    replacing it with `new_timeout`.
+    """
+    orig_timeout = None
+    file_path = os.path.join(ltp_dir, fname)
+
+    with open(file_path, 'r', encoding="utf-8") as c_source:
+        matcher = re.compile(r'\s*.timeout\s*=\s*(\d+).')
+        for line in c_source:
+            match = matcher.search(line)
+            if not match:
+                continue
+
+            timeout = match.group(1)
+            orig_timeout = int(timeout)
+
+    if orig_timeout:
+        if orig_timeout < new_timeout and override:
+            print(f"CHANGE {fname} timeout {orig_timeout} -> {new_timeout}")
+            _sed(file_path, r".timeout = [0-9]*,\n",
+                 f"\t.timeout = {new_timeout},\n")
+        else:
+            print(f"KEEP   {fname} timeout {orig_timeout} (new {new_timeout})")
+    else:
+        print(f"ADD    {fname} timeout {new_timeout}")
+        _sed(file_path,
+             "static struct tst_test test = {",
+             "static struct tst_test test = {\n"
+             f"\t.timeout = {new_timeout},\n")
+
+
+def _patch_all(ltp_dir, timeouts, override):
+    """
+    Patch all tests.
+    """
+    for timeout in timeouts:
+        if timeout['path']:
+            _patch(ltp_dir, timeout['path'], timeout['timeout'], override)
+
+
+def _print_table(timeouts):
+    """
+    Print the timeouts table.
+    """
+    timeouts.sort(key=lambda x: x['timeout'], reverse=True)
+
+    total = 0
+
+    print("Old library tests\n-----------------\n")
+    for timeout in timeouts:
+        if not timeout['newlib']:
+            print(f"{timeout['name']:30s} {timeout['timeout']}")
+            total += 1
+
+    print(f"\n\t{total} tests in total")
+
+    total = 0
+
+    print("\nNew library tests\n-----------------\n")
+    for timeout in timeouts:
+        if timeout['newlib']:
+            print(f"{timeout['name']:30s} {timeout['timeout']}")
+            total += 1
+
+    print(f"\n\t{total} tests in total")
+
+
+def _parse_data(ltp_dir, results_path):
+    """
+    Parse results data and metadata, then it generates timeouts data.
+    """
+    timeouts = []
+    results = None
+    metadata = None
+
+    with open(results_path, 'r', encoding="utf-8") as file:
+        results = json.load(file)
+
+    metadata_path = os.path.join(ltp_dir, 'metadata', 'ltp.json')
+    with open(metadata_path, 'r', encoding="utf-8") as file:
+        metadata = json.load(file)
+
+    for test in results['results']:
+        name = test['test_fqn']
+        duration = test['test']['duration']
+
+        # if test runs for all_filesystems, normalize runtime to one filesystem
+        filesystems = max(1, test['test']['log'].count('TINFO: Formatting /'))
+
+        # check if test is new library test
+        test_is_newlib = name in metadata['tests']
+
+        # store test file path
+        path = None
+        if test_is_newlib:
+            path = metadata['tests'][name]['fname']
+
+        test_has_runtime = False
+        if test_is_newlib:
+            # filter out tests with runtime
+            test_has_runtime = 'runtime' in metadata['tests'][name]
+
+            # timer tests define runtime dynamically in timer library
+            test_has_runtime = 'sample' in metadata['tests'][name]
+
+        # select tests that does not have runtime and which are executed
+        # for a long time
+        if not test_has_runtime and duration >= 0.5:
+            data = {}
+            data["name"] = name
+            data["timeout"] = int(TIMEOUT_MUL * duration/filesystems + 0.5)
+            data["newlib"] = test_is_newlib
+            data["path"] = path
+
+            timeouts.append(data)
+
+    return timeouts
+
+
+def _file_exists(filepath):
+    """
+    Check if the given file path exists.
+    """
+    if not os.path.isfile(filepath):
+        raise argparse.ArgumentTypeError(
+            f"The file '{filepath}' does not exist.")
+    return filepath
+
+
+def _dir_exists(dirpath):
+    """
+    Check if the given directory path exists.
+    """
+    if not os.path.isdir(dirpath):
+        raise argparse.ArgumentTypeError(
+            f"The directory '{dirpath}' does not exist.")
+    return dirpath
+
+
+def run():
+    """
+    Entry point of the script.
+    """
+    parser = argparse.ArgumentParser(
+        description="Script to calculate LTP tests timeouts")
+
+    parser.add_argument(
+        '-l',
+        '--ltp-dir',
+        type=_dir_exists,
+        help='LTP directory',
+        default='/opt/ltp')
+
+    parser.add_argument(
+        '-r',
+        '--results',
+        type=_file_exists,
+        required=True,
+        help='kirk results.json file location')
+
+    parser.add_argument(
+        '-o',
+        '--override',
+        default=False,
+        action='store_true',
+        help='Override test timeouts')
+
+    parser.add_argument(
+        '-p',
+        '--patch',
+        default=False,
+        action='store_true',
+        help='Patch tests with updated timeout')
+
+    parser.add_argument(
+        '-t',
+        '--print-table',
+        default=True,
+        action='store_true',
+        help='Print table with suggested timeouts')
+
+    args = parser.parse_args()
+
+    timeouts = _parse_data(args.ltp_dir, args.results)
+
+    if args.print_table:
+        _print_table(timeouts)
+
+    if args.patch:
+        _patch_all(args.ltp_dir, timeouts, args.override)
+
+
+if __name__ == "__main__":
+    run()

-- 
2.43.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts
  2025-01-22 12:48 ` [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts Andrea Cervesato
@ 2025-01-22 13:13   ` Cyril Hrubis
  0 siblings, 0 replies; 8+ messages in thread
From: Cyril Hrubis @ 2025-01-22 13:13 UTC (permalink / raw)
  To: Andrea Cervesato; +Cc: ltp

Hi!
> This script parses JSON results from kirk and LTP metadata in order
> calculate timeouts for tests based on the result file. It can also patch
> tests automatically.
> 
> The script does:
> 
> - Take the results and pick all tests that run for longer than 0.5s.
>   Multiplie the time with a constant (currently 1.2) to get a suggested
>   timeout.
> 
> - Exclude tests that have runtime defined since these are controlled
>   by the runtime (that filters out all fuzzy sync tests).
> 
>   There is a special case for timer tests that defines runtime only
>   dynamically in the timer library code. This should be possibly fixed
>   with special value for the .runtime in tst_test. E.g.
>   TST_RUNTIME_DYNAMIC for tests that only set runtime in the setup.
> 
> - Normalize the timeout for a single filesystem run if test is running for
>   more than one filesystem.
> 
> - Verify if tests are build on top of old library by checking at
>   metadata file
> 
> - Update test with a  with newly calculated timeout.
>   By default we only increase timeouts but that can be overridden using
>   the -o option.
> 
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> Co-developed-by: Andrea Cervesato <andrea.cervesato@suse.com>
> Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
> ---
>  scripts/calctimeouts.py | 232 ++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 232 insertions(+)
> 
> diff --git a/scripts/calctimeouts.py b/scripts/calctimeouts.py
> new file mode 100755
> index 0000000000000000000000000000000000000000..d5e8fe2c15faad7c8d1ec8c15541f35a97a8c0c4
> --- /dev/null
> +++ b/scripts/calctimeouts.py
> @@ -0,0 +1,232 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +"""
> +This script parses JSON results from kirk and LTP metadata in order to
> +calculate timeouts for tests based on the results file.
> +It can also patch tests automatically and replace the calculated timeout.
> +"""
> +
> +import re
> +import os
> +import json
> +import argparse
> +
> +# The test runtime is multiplied by this to get a timeout
> +TIMEOUT_MUL = 1.2
> +
> +
> +def _sed(fname, expr, replace):
> +    """
> +    Pythonic version of sed command.
> +    """
> +    content = []
> +    matcher = re.compile(expr)
> +
> +    with open(fname, 'r', encoding="utf-8") as data:
> +        for line in data:
> +            match = matcher.search(line)
> +            if not match:
> +                content.append(line)
> +            else:
> +                content.append(replace)
> +
> +    with open(fname, 'w', encoding="utf-8") as data:
> +        data.writelines(content)
> +
> +
> +def _patch(ltp_dir, fname, new_timeout, override):
> +    """
> +    If `override` is True, it patches a test file, searching for timeout and
> +    replacing it with `new_timeout`.

The override is for the cases where the test already has a timeout and
the timeout in the test is bigger than the one we calculated. By default
the function keeps the bigger timeout.

> +    """
> +    orig_timeout = None
> +    file_path = os.path.join(ltp_dir, fname)
> +
> +    with open(file_path, 'r', encoding="utf-8") as c_source:
> +        matcher = re.compile(r'\s*.timeout\s*=\s*(\d+).')
> +        for line in c_source:
> +            match = matcher.search(line)
> +            if not match:
> +                continue
> +
> +            timeout = match.group(1)
> +            orig_timeout = int(timeout)
> +
> +    if orig_timeout:
> +        if orig_timeout < new_timeout and override:
> +            print(f"CHANGE {fname} timeout {orig_timeout} -> {new_timeout}")
> +            _sed(file_path, r".timeout = [0-9]*,\n",
> +                 f"\t.timeout = {new_timeout},\n")
> +        else:
> +            print(f"KEEP   {fname} timeout {orig_timeout} (new {new_timeout})")
> +    else:
> +        print(f"ADD    {fname} timeout {new_timeout}")
> +        _sed(file_path,
> +             "static struct tst_test test = {",
> +             "static struct tst_test test = {\n"
> +             f"\t.timeout = {new_timeout},\n")
> +
> +
> +def _patch_all(ltp_dir, timeouts, override):
> +    """
> +    Patch all tests.
> +    """
> +    for timeout in timeouts:
> +        if timeout['path']:
> +            _patch(ltp_dir, timeout['path'], timeout['timeout'], override)
> +
> +
> +def _print_table(timeouts):
> +    """
> +    Print the timeouts table.
> +    """
> +    timeouts.sort(key=lambda x: x['timeout'], reverse=True)
> +
> +    total = 0
> +
> +    print("Old library tests\n-----------------\n")
> +    for timeout in timeouts:
> +        if not timeout['newlib']:
> +            print(f"{timeout['name']:30s} {timeout['timeout']}")
> +            total += 1
> +
> +    print(f"\n\t{total} tests in total")
> +
> +    total = 0
> +
> +    print("\nNew library tests\n-----------------\n")
> +    for timeout in timeouts:
> +        if timeout['newlib']:
> +            print(f"{timeout['name']:30s} {timeout['timeout']}")
> +            total += 1
> +
> +    print(f"\n\t{total} tests in total")
> +
> +
> +def _parse_data(ltp_dir, results_path):
> +    """
> +    Parse results data and metadata, then it generates timeouts data.
> +    """
> +    timeouts = []
> +    results = None
> +    metadata = None
> +
> +    with open(results_path, 'r', encoding="utf-8") as file:
> +        results = json.load(file)
> +
> +    metadata_path = os.path.join(ltp_dir, 'metadata', 'ltp.json')
> +    with open(metadata_path, 'r', encoding="utf-8") as file:
> +        metadata = json.load(file)
> +
> +    for test in results['results']:
> +        name = test['test_fqn']
> +        duration = test['test']['duration']
> +
> +        # if test runs for all_filesystems, normalize runtime to one filesystem
> +        filesystems = max(1, test['test']['log'].count('TINFO: Formatting /'))
> +
> +        # check if test is new library test
> +        test_is_newlib = name in metadata['tests']
> +
> +        # store test file path
> +        path = None
> +        if test_is_newlib:
> +            path = metadata['tests'][name]['fname']
> +
> +        test_has_runtime = False
> +        if test_is_newlib:
> +            # filter out tests with runtime
> +            test_has_runtime = 'runtime' in metadata['tests'][name]
> +
> +            # timer tests define runtime dynamically in timer library
> +            test_has_runtime = 'sample' in metadata['tests'][name]
> +
> +        # select tests that does not have runtime and which are executed
> +        # for a long time
> +        if not test_has_runtime and duration >= 0.5:
> +            data = {}
> +            data["name"] = name
> +            data["timeout"] = int(TIMEOUT_MUL * duration/filesystems + 0.5)
> +            data["newlib"] = test_is_newlib
> +            data["path"] = path
> +
> +            timeouts.append(data)
> +
> +    return timeouts
> +
> +
> +def _file_exists(filepath):
> +    """
> +    Check if the given file path exists.
> +    """
> +    if not os.path.isfile(filepath):
> +        raise argparse.ArgumentTypeError(
> +            f"The file '{filepath}' does not exist.")
> +    return filepath
> +
> +
> +def _dir_exists(dirpath):
> +    """
> +    Check if the given directory path exists.
> +    """
> +    if not os.path.isdir(dirpath):
> +        raise argparse.ArgumentTypeError(
> +            f"The directory '{dirpath}' does not exist.")
> +    return dirpath
> +
> +
> +def run():
> +    """
> +    Entry point of the script.
> +    """
> +    parser = argparse.ArgumentParser(
> +        description="Script to calculate LTP tests timeouts")
> +
> +    parser.add_argument(
> +        '-l',
> +        '--ltp-dir',
> +        type=_dir_exists,
> +        help='LTP directory',
> +        default='/opt/ltp')

The script is not supposed to be executed from installed tree, it needs
the C source files. So I would argue that '../' or '.' is better
default.

> +    parser.add_argument(
> +        '-r',
> +        '--results',
> +        type=_file_exists,
> +        required=True,
> +        help='kirk results.json file location')
> +
> +    parser.add_argument(
> +        '-o',
> +        '--override',
> +        default=False,
> +        action='store_true',
> +        help='Override test timeouts')
> +
> +    parser.add_argument(
> +        '-p',
> +        '--patch',
> +        default=False,
> +        action='store_true',
> +        help='Patch tests with updated timeout')
> +
> +    parser.add_argument(
> +        '-t',
> +        '--print-table',
> +        default=True,
> +        action='store_true',
> +        help='Print table with suggested timeouts')
> +
> +    args = parser.parse_args()
> +
> +    timeouts = _parse_data(args.ltp_dir, args.results)
> +
> +    if args.print_table:
> +        _print_table(timeouts)
> +
> +    if args.patch:
> +        _patch_all(args.ltp_dir, timeouts, args.override)
> +
> +
> +if __name__ == "__main__":
> +    run()
> 
> -- 
> 2.43.0
> 

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2025-01-22 13:13 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-01-21 12:34 [LTP] [PATCH 0/2] Update test timeouts in an automated way Cyril Hrubis
2025-01-21 12:34 ` [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts Cyril Hrubis
2025-01-21 13:12   ` Andrea Cervesato via ltp
2025-01-21 14:26     ` Cyril Hrubis
2025-01-21 14:32       ` Andrea Cervesato via ltp
2025-01-21 12:34 ` [LTP] [PATCH 2/2] syscalls: Update test timeouts Cyril Hrubis
  -- strict thread matches above, loose matches on Subject: below --
2025-01-22 12:48 [LTP] [PATCH 0/2] Update test timeouts in an automated way Andrea Cervesato
2025-01-22 12:48 ` [LTP] [PATCH 1/2] scripts: Add simple script for calculating timeouts Andrea Cervesato
2025-01-22 13:13   ` Cyril Hrubis

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox