Re: [Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Petr Vorel <petr.vorel@gmail.com>]

Hi Julien,

> The run log of this ltp-testsuite test shows:

>     INFO: runltp script is deprecated, try kirk
>     https://github.com/linux-test-project/kirk

> This commit updates this test to replace this deprecated runltp
> shell script with the newer kirk Python script.

> The logic of this runtime test remains the same: it runs a small number
> of 'read' system call tests, and checks there is no failures and at
> least one test succeed.

Acked-by: Petr Vorel <petr.vorel@gmail.com>

Although this looks to be sufficient for running kirk on SUT (the tested
system), kirk is meant to be run from host (talk: [1], we don't have it
documented nor in kirk nor in LTP, there are just fragments of the discussions
on ML: [2] [3] [4] [5]). In short reliable kernel testing requires test to be
executed outside of the host (some tests influence SUT a lot).

Therefore kirk can connect to SUT via ssh, run qcow image via QEMU [6], or
experimental LTX protocol [7] [2] [3].

Due that I was thinking to create kirk and ltx Buildroot host packages.

Kind regards,
Petr

[1] https://www.youtube.com/watch?v=T1ImuNr9Oxo
[2] https://lore.kernel.org/ltp/ZmM25L0EmJufsS-f@yuki/
[3] https://lore.kernel.org/ltp/ZmbFyjuXndeXCLp8@rei/
[4] https://lore.kernel.org/ltp/ZmMrBnkIXKfrF8Xv@yuki/
[5] https://lore.kernel.org/ltp/ZmM5y3a4xNCsRTT-@yuki/
[6] https://kirk.readthedocs.io/en/latest/users/quickstart.html
[7] https://github.com/linux-test-project/ltx

> Cc: Petr Vorel <petr.vorel@gmail.com>
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/12510635102
> ---
>  .../tests/package/test_ltp_testsuite.py       | 33 +++++++++++--------
>  1 file changed, 20 insertions(+), 13 deletions(-)

> diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
> index a85ceb35a6..6ff59b9700 100644
> --- a/support/testing/tests/package/test_ltp_testsuite.py
> +++ b/support/testing/tests/package/test_ltp_testsuite.py
> @@ -1,3 +1,4 @@
> +import json
>  import os

>  import infra.basetest
> @@ -7,6 +8,7 @@ class TestLtpTestsuite(infra.basetest.BRTest):
>      config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
>          """
>          BR2_PACKAGE_LTP_TESTSUITE=y
> +        BR2_PACKAGE_PYTHON3=y
>          BR2_TARGET_ROOTFS_EXT2=y
>          BR2_TARGET_ROOTFS_EXT2_4=y
>          BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
> @@ -21,19 +23,24 @@ class TestLtpTestsuite(infra.basetest.BRTest):
>                             options=["-drive", f"file={drive},if=scsi,format=raw"])
>          self.emulator.login()

> +        ltp_root = "/usr/lib/ltp-testsuite"
> +        report_file = "/tmp/ltp-report.json"
> +
>          # We run a reduced number of tests (read syscall tests) for a
> -        # fast execution. See "runltp --help" for option details.
> -        cmd = "/usr/lib/ltp-testsuite/runltp"
> -        cmd += " -p -q"
> -        cmd += " -s ^read0[0-9]*"
> -        cmd += " -l /tmp/ltp.log"
> -        cmd += " -o /tmp/ltp.output"
> -        cmd += " -C /tmp/ltp.failed"
> -        cmd += " -T /tmp/ltp.tconf"
> -        self.assertRunOk(cmd)
> +        # fast execution. See "kirk --help" for option details.
> +        cmd = f"LTPROOT={ltp_root}"
> +        cmd += f" {ltp_root}/kirk"
> +        cmd += " --sut host"
> +        cmd += " --framework ltp"
> +        cmd += " --run-suite syscalls"
> +        cmd += " --run-pattern '^read0[0-9]*'"
> +        cmd += f" --json-report {report_file}"
> +        self.assertRunOk(cmd, timeout=30)

> -        # We print the LTP run log and check there was zero failure in
> -        # our test selection.
> -        out, ret = self.emulator.run("cat /tmp/ltp.log")
> +        # We print the LTP report and check there was at least one
> +        # passed test and zero failure in our selection.
> +        out, ret = self.emulator.run(f"cat {report_file} && echo")
>          self.assertEqual(ret, 0)
> -        self.assertIn("Total Failures: 0", out)
> +        report = json.loads("".join(out))
> +        self.assertEqual(report['stats']['failed'], 0)
> +        self.assertGreater(report['stats']['passed'], 0)
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot