Re: [LTP] [Automated-testing] [PATCH v2 00/30] Introduce runtime and conver tests

[Richard Palethorpe <rpalethorpe@suse.de>]

Hello, Cyril,

"Cyril Hrubis" <chrubis@suse.cz> writes:

> This patchset introduces a concept of runtime, which is described well
> enough in the first patch of the series and converts all tests that were
> previously using timeout API to runtime API.
>
> The main change to the v1 of the patchset is that the runtime is now per
> iteration, e.g. if we run the test for all_filesystem the runtime
> applies to a single filesystem only. And at the same time the overall
> test timeout is a sum of static timeout and runtime. The static part of
> the timeout is meant to cover test teardown and any inaccuracy in the
> test runtime accounting.
>
> The patches that convert tests to runtime API fall into these classes:
>
> - fzsync tests
> - faster timeout tests
> - I/O tests
> - timer precision tests
> - oom tests
> - misc tests that run for some time (controlled by command line
>   parameter)
>
> fzsync tests
> ============
>
> These tests previously defined their runtime based on scale factor that
> used slice of the default timeout (300s) for the test runtime.
>
> Now the runtime is explicitly encoded in the test, which I find much
> more user friendly because it's clear how long will the test run.
>
> faster timeout tests
> ====================
>
> There were several tests that set low timeout becuase the test may lock
> up. In these cases the timeout was simply removed because the default
> value (30s) should be small enough.
>
> I/O tests
> =========
>
> The aiodio tests were modified so that their runtime is capped to
> 30 minutes by default. That means that even on very slow machine they
> finish in a reasonable time.
>
> timer precision tests
> =====================
>
> These test simply set the test timeout to be 110% of the sum of the
> sleep_time * samples. That is to accomodate for the context switches and
> upkeeping between the calls in the timer library.
>
>
> oom tests
> =========
>
> Here we just set the .max_iteration_runtime to TST_RUNTIME_UNLIMITED,
> which has the very same effect as .timeout = -1 previously had.
>
> misc test
> =========
>
> In this case the test code was simplified quite a lot since the runtime
> is now passed in a unified way and parsed in the test library.
>
> TODO:
>
> This patchset is reasonably complete in a sense that it removes the
> timeout API at the end. Still there are a few things to consider:
>
> - is reusing the -I parameter a good idea? Wouldn't adding new parameter
>   (-r) be better?

Perhaps we could deprecate -I, but convert it to use the new mechanism
while trying to keep the behaviour similar.

I'm not sure we need a global -r option, but unlike -i it's not much
effort to support. It would not surprise me if there are existing tests
which don't work with -i.

>
> - there are quite likely tests that run for more than a second or a two
>   and should be made runtime aware
>
> - anything else?
>
> Cyril Hrubis (30):
>   Introduce a concept of max runtime
>   mtest06/mmap1: Convert to runtime
>   mtest06/mmap3: Convert to runtime
>   mtest01/mtest01: Convert to runtime
>   cve/cve-2015-3290: Convert to runtime
>   crypto/af_alg02: Convert to runtime.
>   crypto/pcrypt_aead01: Convert to runtime
>   syscalls/clock_gettime01: Remove useless timeout
>   syscalls/fanotify22: Remove useless timeout
>   syscalls/gettimeofday02: Convert to runtime.
>   syscalls/inotify06: Convert to runtime.
>   syscalls/inotify01: Remove now useless timeout
>   syscalls/perf_event_open03: Convert to runtime
>   syscalls/readv01: Remove now useless timeout
>   syscalls/tgkill03: Remove now unused timeout
>   syscalls/setsockopt09: Remove now useless timeout
>   syscalls/userfaultfd01: Remove now useless timeout
>   syscalls/move_pages12: Convert to runtime
>   syscalls/rt_sigqueueinfo01: Remove now useless timeout
>   mem/mallocstress: Convert to runtime
>   mem/{oom,min_free_kbytes}: Convert to runtime
>   crypto/af_alg02: Convert to runtime
>   fuzzy_sync: Convert to runtime
>   ltp-aiodio/dio_sparse,aiodio_sparse: Convert to runtime.
>   ltp-aiodio/read_checkzero: Remove
>   ltp-aiodio/dio_{truncate,append}: Convert to runtime
>   ltp-aiodio/dio_read: Convert to runtime
>   ltp-aiodio/aiodio_append: Convert to runtime
>   timer_test: Convert to runtime
>   tst_test: Remove timeout stubs
>
>  doc/user-guide.txt                            |   4 +
>  include/tst_fuzzy_sync.h                      |  19 +--
>  include/tst_test.h                            |  31 +++-
>  lib/newlib_tests/.gitignore                   |   5 +-
>  lib/newlib_tests/test10.c                     |  22 ---
>  lib/newlib_tests/test12.c                     |  21 ---
>  lib/newlib_tests/test13.c                     |   1 -
>  lib/newlib_tests/test_children_cleanup.c      |   1 -
>  .../{test18.c => test_runtime01.c}            |  10 +-
>  lib/newlib_tests/test_runtime02.c             |  31 ++++
>  lib/tst_test.c                                | 144 +++++++++++++-----
>  lib/tst_timer_test.c                          |  12 +-
>  runtest/ltp-aiodio.part4                      |   2 -
>  testcases/cve/cve-2015-3290.c                 |   8 +-
>  testcases/kernel/crypto/af_alg02.c            |   4 +-
>  testcases/kernel/crypto/af_alg07.c            |   1 +
>  testcases/kernel/crypto/pcrypt_aead01.c       |   3 +-
>  testcases/kernel/io/ltp-aiodio/.gitignore     |   1 -
>  .../kernel/io/ltp-aiodio/aiodio_append.c      |  25 ++-
>  .../kernel/io/ltp-aiodio/aiodio_sparse.c      |  26 +++-
>  testcases/kernel/io/ltp-aiodio/common.h       |  12 +-
>  testcases/kernel/io/ltp-aiodio/dio_append.c   |   4 +
>  testcases/kernel/io/ltp-aiodio/dio_read.c     |   9 ++
>  testcases/kernel/io/ltp-aiodio/dio_sparse.c   |   6 +-
>  testcases/kernel/io/ltp-aiodio/dio_truncate.c |   6 +
>  .../kernel/io/ltp-aiodio/read_checkzero.c     |  84 ----------
>  testcases/kernel/mem/mtest01/mtest01.c        |  14 +-
>  testcases/kernel/mem/mtest06/mmap1.c          |  24 +--
>  testcases/kernel/mem/mtest06/mmap3.c          |  12 +-
>  testcases/kernel/mem/mtest07/mallocstress.c   |  11 +-
>  testcases/kernel/mem/oom/oom01.c              |   2 +-
>  testcases/kernel/mem/oom/oom02.c              |   2 +-
>  testcases/kernel/mem/oom/oom03.c              |   2 +-
>  testcases/kernel/mem/oom/oom04.c              |   2 +-
>  testcases/kernel/mem/oom/oom05.c              |   2 +-
>  testcases/kernel/mem/thp/thp04.c              |   1 +
>  .../kernel/mem/tunable/min_free_kbytes.c      |   2 +-
>  testcases/kernel/pty/pty03.c                  |   2 +-
>  testcases/kernel/pty/pty05.c                  |   1 +
>  testcases/kernel/pty/pty06.c                  |   1 +
>  testcases/kernel/pty/pty07.c                  |   1 +
>  testcases/kernel/sound/snd_seq01.c            |   2 +-
>  testcases/kernel/sound/snd_timer01.c          |   1 +
>  testcases/kernel/syscalls/bind/bind06.c       |   2 +-
>  .../syscalls/clock_gettime/clock_gettime01.c  |   1 -
>  .../kernel/syscalls/fanotify/fanotify22.c     |   1 -
>  .../syscalls/gettimeofday/gettimeofday02.c    |  15 +-
>  testcases/kernel/syscalls/inotify/inotify06.c |   7 +-
>  testcases/kernel/syscalls/inotify/inotify09.c |   1 +
>  testcases/kernel/syscalls/inotify/inotify11.c |   1 -
>  .../kernel/syscalls/ipc/shmctl/shmctl05.c     |   2 +-
>  .../kernel/syscalls/move_pages/move_pages12.c |   4 +-
>  .../perf_event_open/perf_event_open03.c       |  15 +-
>  testcases/kernel/syscalls/readv/readv01.c     |   1 -
>  .../rt_sigqueueinfo/rt_sigqueueinfo01.c       |   1 -
>  testcases/kernel/syscalls/sendmsg/sendmsg03.c |   1 +
>  .../kernel/syscalls/setsockopt/setsockopt06.c |   2 +-
>  .../kernel/syscalls/setsockopt/setsockopt07.c |   1 +
>  .../kernel/syscalls/setsockopt/setsockopt09.c |   1 -
>  testcases/kernel/syscalls/tgkill/tgkill03.c   |   1 -
>  .../syscalls/timerfd/timerfd_settime02.c      |   1 +
>  .../syscalls/userfaultfd/userfaultfd01.c      |   1 -
>  testcases/kernel/syscalls/writev/writev03.c   |   2 +-
>  63 files changed, 339 insertions(+), 296 deletions(-)
>  delete mode 100644 lib/newlib_tests/test10.c
>  delete mode 100644 lib/newlib_tests/test12.c
>  rename lib/newlib_tests/{test18.c => test_runtime01.c} (58%)
>  create mode 100644 lib/newlib_tests/test_runtime02.c
>  delete mode 100644 testcases/kernel/io/ltp-aiodio/read_checkzero.c


-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp