Re: [Buildroot] [PATCH 1/1] support/testing: new ltp-testsuite runtime test

[Petr Vorel <petr.vorel@gmail.com>]

Hi Julien,

> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/8388435290
> ---
>  DEVELOPERS                                    |  1 +
>  .../tests/package/test_ltp_testsuite.py       | 39 +++++++++++++++++++
>  2 files changed, 40 insertions(+)
>  create mode 100644 support/testing/tests/package/test_ltp_testsuite.py

> diff --git a/DEVELOPERS b/DEVELOPERS
> index 3945803d53..1ff269b362 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -1931,6 +1931,7 @@ F:	support/testing/tests/package/test_links.py
>  F:	support/testing/tests/package/test_links/
>  F:	support/testing/tests/package/test_lrzip.py
>  F:	support/testing/tests/package/test_lrzsz.py
> +F:	support/testing/tests/package/test_ltp_testsuite.py
>  F:	support/testing/tests/package/test_ltrace.py
>  F:	support/testing/tests/package/test_lvm2.py
>  F:	support/testing/tests/package/test_lzip.py
> diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
> new file mode 100644
> index 0000000000..a85ceb35a6
> --- /dev/null
> +++ b/support/testing/tests/package/test_ltp_testsuite.py
> @@ -0,0 +1,39 @@
> +import os
> +
> +import infra.basetest
> +
> +
> +class TestLtpTestsuite(infra.basetest.BRTest):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        BR2_PACKAGE_LTP_TESTSUITE=y

Thanks a lot for taking care about LTP. Feel free to Cc me on LTP related
patches and issues.

> +        BR2_TARGET_ROOTFS_EXT2=y
> +        BR2_TARGET_ROOTFS_EXT2_4=y
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        """
> +
> +    def test_run(self):
> +        drive = os.path.join(self.builddir, "images", "rootfs.ext4")
> +        self.emulator.boot(arch="armv5",
> +                           kernel="builtin",
> +                           kernel_cmdline=["rootwait", "root=/dev/sda"],
> +                           options=["-drive", f"file={drive},if=scsi,format=raw"])
> +        self.emulator.login()
> +
> +        # We run a reduced number of tests (read syscall tests) for a
> +        # fast execution. See "runltp --help" for option details.
> +        cmd = "/usr/lib/ltp-testsuite/runltp"
> +        cmd += " -p -q"
> +        cmd += " -s ^read0[0-9]*"
> +        cmd += " -l /tmp/ltp.log"
> +        cmd += " -o /tmp/ltp.output"
> +        cmd += " -C /tmp/ltp.failed"
> +        cmd += " -T /tmp/ltp.tconf"

While this above is OK, some notes:

1) Many LTP tests require LTPROOT environment variable (to access some data files),
e.g.:

export LTPROOT="/usr/lib/ltp-testsuite"

See https://linux-test-project.readthedocs.io/en/latest/users/setup_tests.html

It looks to me that infra.emulator.Emulator does not support inserting
environment variables via os.environ.

2) runltp old script has been deprecated, kirk is the replacement:

https://github.com/linux-test-project/kirk

Currently kirk supports few testing frameworks: LTP, kselftests and liburing.

While kirk can be run o SUT (like runltp), the best usage is to run it on the
host. It supports testing via starting QEMU VM, SSH and not yet stable LTX
backend.

Due the above I'd like to add kirk host package to Buildroot.

Kind regards,
Petr


> +        self.assertRunOk(cmd)
> +
> +        # We print the LTP run log and check there was zero failure in
> +        # our test selection.
> +        out, ret = self.emulator.run("cat /tmp/ltp.log")
> +        self.assertEqual(ret, 0)
> +        self.assertIn("Total Failures: 0", out)
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot