[Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Julien Olivain via buildroot]

The run log of this ltp-testsuite test shows:

    INFO: runltp script is deprecated, try kirk
    https://github.com/linux-test-project/kirk

This commit updates this test to replace this deprecated runltp
shell script with the newer kirk Python script.

The logic of this runtime test remains the same: it runs a small number
of 'read' system call tests, and checks there is no failures and at
least one test succeed.

Cc: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Julien Olivain <ju.o@free.fr>
---
Patch tested in:
https://gitlab.com/jolivain/buildroot/-/jobs/12510635102
---
 .../tests/package/test_ltp_testsuite.py       | 33 +++++++++++--------
 1 file changed, 20 insertions(+), 13 deletions(-)

diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
index a85ceb35a6..6ff59b9700 100644
--- a/support/testing/tests/package/test_ltp_testsuite.py
+++ b/support/testing/tests/package/test_ltp_testsuite.py
@@ -1,3 +1,4 @@
+import json
 import os
 
 import infra.basetest
@@ -7,6 +8,7 @@ class TestLtpTestsuite(infra.basetest.BRTest):
     config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
         """
         BR2_PACKAGE_LTP_TESTSUITE=y
+        BR2_PACKAGE_PYTHON3=y
         BR2_TARGET_ROOTFS_EXT2=y
         BR2_TARGET_ROOTFS_EXT2_4=y
         BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
@@ -21,19 +23,24 @@ class TestLtpTestsuite(infra.basetest.BRTest):
                            options=["-drive", f"file={drive},if=scsi,format=raw"])
         self.emulator.login()
 
+        ltp_root = "/usr/lib/ltp-testsuite"
+        report_file = "/tmp/ltp-report.json"
+
         # We run a reduced number of tests (read syscall tests) for a
-        # fast execution. See "runltp --help" for option details.
-        cmd = "/usr/lib/ltp-testsuite/runltp"
-        cmd += " -p -q"
-        cmd += " -s ^read0[0-9]*"
-        cmd += " -l /tmp/ltp.log"
-        cmd += " -o /tmp/ltp.output"
-        cmd += " -C /tmp/ltp.failed"
-        cmd += " -T /tmp/ltp.tconf"
-        self.assertRunOk(cmd)
+        # fast execution. See "kirk --help" for option details.
+        cmd = f"LTPROOT={ltp_root}"
+        cmd += f" {ltp_root}/kirk"
+        cmd += " --sut host"
+        cmd += " --framework ltp"
+        cmd += " --run-suite syscalls"
+        cmd += " --run-pattern '^read0[0-9]*'"
+        cmd += f" --json-report {report_file}"
+        self.assertRunOk(cmd, timeout=30)
 
-        # We print the LTP run log and check there was zero failure in
-        # our test selection.
-        out, ret = self.emulator.run("cat /tmp/ltp.log")
+        # We print the LTP report and check there was at least one
+        # passed test and zero failure in our selection.
+        out, ret = self.emulator.run(f"cat {report_file} && echo")
         self.assertEqual(ret, 0)
-        self.assertIn("Total Failures: 0", out)
+        report = json.loads("".join(out))
+        self.assertEqual(report['stats']['failed'], 0)
+        self.assertGreater(report['stats']['passed'], 0)
-- 
2.52.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Petr Vorel]

Hi Julien,

> The run log of this ltp-testsuite test shows:

>     INFO: runltp script is deprecated, try kirk
>     https://github.com/linux-test-project/kirk

> This commit updates this test to replace this deprecated runltp
> shell script with the newer kirk Python script.

> The logic of this runtime test remains the same: it runs a small number
> of 'read' system call tests, and checks there is no failures and at
> least one test succeed.

Acked-by: Petr Vorel <petr.vorel@gmail.com>

Although this looks to be sufficient for running kirk on SUT (the tested
system), kirk is meant to be run from host (talk: [1], we don't have it
documented nor in kirk nor in LTP, there are just fragments of the discussions
on ML: [2] [3] [4] [5]). In short reliable kernel testing requires test to be
executed outside of the host (some tests influence SUT a lot).

Therefore kirk can connect to SUT via ssh, run qcow image via QEMU [6], or
experimental LTX protocol [7] [2] [3].

Due that I was thinking to create kirk and ltx Buildroot host packages.

Kind regards,
Petr

[1] https://www.youtube.com/watch?v=T1ImuNr9Oxo
[2] https://lore.kernel.org/ltp/ZmM25L0EmJufsS-f@yuki/
[3] https://lore.kernel.org/ltp/ZmbFyjuXndeXCLp8@rei/
[4] https://lore.kernel.org/ltp/ZmMrBnkIXKfrF8Xv@yuki/
[5] https://lore.kernel.org/ltp/ZmM5y3a4xNCsRTT-@yuki/
[6] https://kirk.readthedocs.io/en/latest/users/quickstart.html
[7] https://github.com/linux-test-project/ltx

> Cc: Petr Vorel <petr.vorel@gmail.com>
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/12510635102
> ---
>  .../tests/package/test_ltp_testsuite.py       | 33 +++++++++++--------
>  1 file changed, 20 insertions(+), 13 deletions(-)

> diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
> index a85ceb35a6..6ff59b9700 100644
> --- a/support/testing/tests/package/test_ltp_testsuite.py
> +++ b/support/testing/tests/package/test_ltp_testsuite.py
> @@ -1,3 +1,4 @@
> +import json
>  import os

>  import infra.basetest
> @@ -7,6 +8,7 @@ class TestLtpTestsuite(infra.basetest.BRTest):
>      config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
>          """
>          BR2_PACKAGE_LTP_TESTSUITE=y
> +        BR2_PACKAGE_PYTHON3=y
>          BR2_TARGET_ROOTFS_EXT2=y
>          BR2_TARGET_ROOTFS_EXT2_4=y
>          BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
> @@ -21,19 +23,24 @@ class TestLtpTestsuite(infra.basetest.BRTest):
>                             options=["-drive", f"file={drive},if=scsi,format=raw"])
>          self.emulator.login()

> +        ltp_root = "/usr/lib/ltp-testsuite"
> +        report_file = "/tmp/ltp-report.json"
> +
>          # We run a reduced number of tests (read syscall tests) for a
> -        # fast execution. See "runltp --help" for option details.
> -        cmd = "/usr/lib/ltp-testsuite/runltp"
> -        cmd += " -p -q"
> -        cmd += " -s ^read0[0-9]*"
> -        cmd += " -l /tmp/ltp.log"
> -        cmd += " -o /tmp/ltp.output"
> -        cmd += " -C /tmp/ltp.failed"
> -        cmd += " -T /tmp/ltp.tconf"
> -        self.assertRunOk(cmd)
> +        # fast execution. See "kirk --help" for option details.
> +        cmd = f"LTPROOT={ltp_root}"
> +        cmd += f" {ltp_root}/kirk"
> +        cmd += " --sut host"
> +        cmd += " --framework ltp"
> +        cmd += " --run-suite syscalls"
> +        cmd += " --run-pattern '^read0[0-9]*'"
> +        cmd += f" --json-report {report_file}"
> +        self.assertRunOk(cmd, timeout=30)

> -        # We print the LTP run log and check there was zero failure in
> -        # our test selection.
> -        out, ret = self.emulator.run("cat /tmp/ltp.log")
> +        # We print the LTP report and check there was at least one
> +        # passed test and zero failure in our selection.
> +        out, ret = self.emulator.run(f"cat {report_file} && echo")
>          self.assertEqual(ret, 0)
> -        self.assertIn("Total Failures: 0", out)
> +        report = json.loads("".join(out))
> +        self.assertEqual(report['stats']['failed'], 0)
> +        self.assertGreater(report['stats']['passed'], 0)
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Thomas Petazzoni via buildroot]

On Sun, 21 Dec 2025 19:08:10 +0100
Julien Olivain via buildroot <buildroot@buildroot.org> wrote:

> The run log of this ltp-testsuite test shows:
> 
>     INFO: runltp script is deprecated, try kirk
>     https://github.com/linux-test-project/kirk
> 
> This commit updates this test to replace this deprecated runltp
> shell script with the newer kirk Python script.
> 
> The logic of this runtime test remains the same: it runs a small number
> of 'read' system call tests, and checks there is no failures and at
> least one test succeed.
> 
> Cc: Petr Vorel <petr.vorel@gmail.com>
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/12510635102
> ---
>  .../tests/package/test_ltp_testsuite.py       | 33 +++++++++++--------
>  1 file changed, 20 insertions(+), 13 deletions(-)

Thanks, I've applied. However, it's a bit of a pity that now Python 3
is needed on the target to run LTP.

One option would as suggested by Petr to use kirk in its intended mode,
with kirk running on the host, and connected over SSH to the target to
run tests.

On the other hand, it was a lot more simple/convenient to be able to
just run a simple shell script on the target to run the test suite.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Petr Vorel]

Hi Thomas,

> On Sun, 21 Dec 2025 19:08:10 +0100
> Julien Olivain via buildroot <buildroot@buildroot.org> wrote:

> > The run log of this ltp-testsuite test shows:

> >     INFO: runltp script is deprecated, try kirk
> >     https://github.com/linux-test-project/kirk

> > This commit updates this test to replace this deprecated runltp
> > shell script with the newer kirk Python script.

> > The logic of this runtime test remains the same: it runs a small number
> > of 'read' system call tests, and checks there is no failures and at
> > least one test succeed.

> > Cc: Petr Vorel <petr.vorel@gmail.com>
> > Signed-off-by: Julien Olivain <ju.o@free.fr>
> > ---
> > Patch tested in:
> > https://gitlab.com/jolivain/buildroot/-/jobs/12510635102
> > ---
> >  .../tests/package/test_ltp_testsuite.py       | 33 +++++++++++--------
> >  1 file changed, 20 insertions(+), 13 deletions(-)

> Thanks, I've applied.
+1, thanks!

> However, it's a bit of a pity that now Python 3
> is needed on the target to run LTP.

Yeah. I don't like it either. Specially LTP tests themselves does not require
anything (single test is either binary written in C statically linked LTP
libraries, we also have few POSIX compatible shell based tests which require
barely minimum of external dependencies), it can be run as is.

ATM we have list of tests in so called runtest files [1], which contain list of
tests (each line is single test: "test_label binary/script optional_arguments"),
therefore anything which can remove the first parameter and execute the rest can
be LTP runner. i.e. Buildroot custom script could be really minimal.

runltp does quite a few other things but these who need them should really move
to kirk. Also we plan to implement in kirk and ltx running tests in parallel.
That will save a lot of runtime.

[1] https://github.com/linux-test-project/ltp/tree/master/runtest

> One option would as suggested by Petr to use kirk in its intended mode,
> with kirk running on the host, and connected over SSH to the target to
> run tests.

> On the other hand, it was a lot more simple/convenient to be able to
> just run a simple shell script on the target to run the test suite.

Would it be possible to have both? Vendor runltp for local use and package kirk
as a host package (+ ltx as normal package)?

Kind regards,
Petr

> Thomas
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Arnout Vandecappelle via buildroot]

In reply of:
> The run log of this ltp-testsuite test shows:
> 
>     INFO: runltp script is deprecated, try kirk
>     https://github.com/linux-test-project/kirk
> 
> This commit updates this test to replace this deprecated runltp
> shell script with the newer kirk Python script.
> 
> The logic of this runtime test remains the same: it runs a small number
> of 'read' system call tests, and checks there is no failures and at
> least one test succeed.
> 
> Cc: Petr Vorel <petr.vorel@gmail.com>
> Signed-off-by: Julien Olivain <ju.o@free.fr>

Applied to 2025.02.x and 2025.11.x. Thanks

> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/12510635102
> ---
>  .../tests/package/test_ltp_testsuite.py       | 33 +++++++++++--------
>  1 file changed, 20 insertions(+), 13 deletions(-)
> 
> diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
> index a85ceb35a6..6ff59b9700 100644
> --- a/support/testing/tests/package/test_ltp_testsuite.py
> +++ b/support/testing/tests/package/test_ltp_testsuite.py
> @@ -1,3 +1,4 @@
> +import json
>  import os
>  
>  import infra.basetest
> @@ -7,6 +8,7 @@ class TestLtpTestsuite(infra.basetest.BRTest):
>      config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
>          """
>          BR2_PACKAGE_LTP_TESTSUITE=y
> +        BR2_PACKAGE_PYTHON3=y
>          BR2_TARGET_ROOTFS_EXT2=y
>          BR2_TARGET_ROOTFS_EXT2_4=y
>          BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
> @@ -21,19 +23,24 @@ class TestLtpTestsuite(infra.basetest.BRTest):
>                             options=["-drive", f"file={drive},if=scsi,format=raw"])
>          self.emulator.login()
>  
> +        ltp_root = "/usr/lib/ltp-testsuite"
> +        report_file = "/tmp/ltp-report.json"
> +
>          # We run a reduced number of tests (read syscall tests) for a
> -        # fast execution. See "runltp --help" for option details.
> -        cmd = "/usr/lib/ltp-testsuite/runltp"
> -        cmd += " -p -q"
> -        cmd += " -s ^read0[0-9]*"
> -        cmd += " -l /tmp/ltp.log"
> -        cmd += " -o /tmp/ltp.output"
> -        cmd += " -C /tmp/ltp.failed"
> -        cmd += " -T /tmp/ltp.tconf"
> -        self.assertRunOk(cmd)
> +        # fast execution. See "kirk --help" for option details.
> +        cmd = f"LTPROOT={ltp_root}"
> +        cmd += f" {ltp_root}/kirk"
> +        cmd += " --sut host"
> +        cmd += " --framework ltp"
> +        cmd += " --run-suite syscalls"
> +        cmd += " --run-pattern '^read0[0-9]*'"
> +        cmd += f" --json-report {report_file}"
> +        self.assertRunOk(cmd, timeout=30)
>  
> -        # We print the LTP run log and check there was zero failure in
> -        # our test selection.
> -        out, ret = self.emulator.run("cat /tmp/ltp.log")
> +        # We print the LTP report and check there was at least one
> +        # passed test and zero failure in our selection.
> +        out, ret = self.emulator.run(f"cat {report_file} && echo")
>          self.assertEqual(ret, 0)
> -        self.assertIn("Total Failures: 0", out)
> +        report = json.loads("".join(out))
> +        self.assertEqual(report['stats']['failed'], 0)
> +        self.assertGreater(report['stats']['passed'], 0)
> -- 
> 2.52.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Thomas Petazzoni via buildroot]

Hello,

On Tue, Dec 30, 2025 at 06:49:58PM +0100, Petr Vorel wrote:

> Yeah. I don't like it either. Specially LTP tests themselves does not require
> anything (single test is either binary written in C statically linked LTP
> libraries, we also have few POSIX compatible shell based tests which require
> barely minimum of external dependencies), it can be run as is.
> 
> ATM we have list of tests in so called runtest files [1], which contain list of
> tests (each line is single test: "test_label binary/script optional_arguments"),
> therefore anything which can remove the first parameter and execute the rest can
> be LTP runner. i.e. Buildroot custom script could be really minimal.

I don't think I would want Buildroot to carry its own custom script
for this.

> runltp does quite a few other things but these who need them should really move
> to kirk. Also we plan to implement in kirk and ltx running tests in parallel.
> That will save a lot of runtime.
> 
> [1] https://github.com/linux-test-project/ltp/tree/master/runtest
> 
> > One option would as suggested by Petr to use kirk in its intended mode,
> > with kirk running on the host, and connected over SSH to the target to
> > run tests.
> 
> > On the other hand, it was a lot more simple/convenient to be able to
> > just run a simple shell script on the target to run the test suite.
> 
> Would it be possible to have both? Vendor runltp for local use and package kirk
> as a host package (+ ltx as normal package)?

I'm not sure what you're suggesting as "both"? Do you mean, having a
simple test runner that we can run directly on the target, but also
offer the possibility of building/installing kirk to run the test
suite from the host?

I'm not sure what ltx is in this context.

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: ltp-testsuite: replace runltp by kirk

[Petr Vorel]

Hi all,

> Hello,

[ Cc Maxim ]

> On Tue, Dec 30, 2025 at 06:49:58PM +0100, Petr Vorel wrote:

> > Yeah. I don't like it either. Specially LTP tests themselves does not require
> > anything (single test is either binary written in C statically linked LTP
> > libraries, we also have few POSIX compatible shell based tests which require
> > barely minimum of external dependencies), it can be run as is.

> > ATM we have list of tests in so called runtest files [1], which contain list of
> > tests (each line is single test: "test_label binary/script optional_arguments"),
> > therefore anything which can remove the first parameter and execute the rest can
> > be LTP runner. i.e. Buildroot custom script could be really minimal.

> I don't think I would want Buildroot to carry its own custom script
> for this.

Fair enough.

We suggested the solution for the people which insisted we'd keep old runltp
script forever.

And I tend to suggest it when see patches like this one:

https://lore.kernel.org/buildroot/20260114192413.569151-1-tarelovma@yandex.ru/

> > runltp does quite a few other things but these who need them should really move
> > to kirk. Also we plan to implement in kirk and ltx running tests in parallel.
> > That will save a lot of runtime.

> > [1] https://github.com/linux-test-project/ltp/tree/master/runtest

> > > One option would as suggested by Petr to use kirk in its intended mode,
> > > with kirk running on the host, and connected over SSH to the target to
> > > run tests.

> > > On the other hand, it was a lot more simple/convenient to be able to
> > > just run a simple shell script on the target to run the test suite.

> > Would it be possible to have both? Vendor runltp for local use and package kirk
> > as a host package (+ ltx as normal package)?

> I'm not sure what you're suggesting as "both"? Do you mean, having a
> simple test runner that we can run directly on the target, but also
> offer the possibility of building/installing kirk to run the test
> suite from the host?

Yes.

> I'm not sure what ltx is in this context.

https://github.com/linux-test-project/ltx
(ltx is not yet relevant because it is still experimental / unfinished.
It's a test executor which allows to execute binaries on host in the fastest way
as possible. In the future together with LTP metadata it will allow to run tests
in parallel - speedup.)

Kind regards,
Petr

> Best regards,

> Thomas
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot