Re: [OE-core] [PATCH 2/2] oeqa: replace runltp with kirk

["David Nyström" <david.nystrom@est.tech>]

[-- Attachment #1: Type: text/plain, Size: 10715 bytes --]


On Thu, 26 Mar 2026, Daniel Turull via lists.openembedded.org wrote:

> Just an update.
>
> It is taking a bit more time than expected. My original test only covered the math test suite in ltp to verify the functionality.
>
> Some of the ltp test are failing with OOM, some related to systemd-udev. I'll be disabling them now to have a working ltp. Then look at the underlaying issues.
>
> So far I have found 4 failing test cases. Using the default config without any changes with qemux86-64
>
> min_free_kbytes (mm) (OOM)

This seems to be a buggy testcase for automation. Disable until its fixed. 
Even if the test own memory consumers sets their own OOM-score higher, 
there are still chances that the OOM-killer kill the wrong things.

> pty07 (pty) (OOM)
> ptem02 (pty) (OOM)

These seem to me to be related to systemd-udevd having unbounded message 
queue sizes to its udev workers. IMO, this is a systemd-udevd issue. 
Some other udev implementations starts dropping events.

We should have seen this when using runltp as well ?

Death spiral: systemd_259.5 
1. Test tight loop creating devices floods systemd-udevd's workers 
unbounded inbox queue(s). 
- Single core, low mem, and long running udev rules makes the problem 
worse.
2. systemd-udevd grows to consume all available RAM
3. OOM killer kills everything but systemd-udevd (OOMScoreAdjust=-1000)
4. Kernel panic: "no killable processes"


> cve-2018-13405 (cve) (deadlock kernel panic)
>
> Giving enough memory 16GB makes them go away but I'm trying to make it work with 4GB of RAM.
>
> Best regards,
> Daniel
>
>> -----Original Message-----
>> From: openembedded-core@lists.openembedded.org <openembedded-
>> core@lists.openembedded.org> On Behalf Of Richard Purdie via
>> lists.openembedded.org
>> Sent: Wednesday, 25 March 2026 17:36
>> To: Daniel Turull <daniel.turull@ericsson.com>; openembedded-
>> core@lists.openembedded.org
>> Cc: pratik.farkase@est.tech
>> Subject: Re: [OE-core] [PATCH 2/2] oeqa: replace runltp with kirk
>>
>> On Wed, 2026-03-25 at 09:40 +0100, daniel.turull@ericsson.com wrote:
>>> From: Daniel Turull <daniel.turull@ericsson.com>
>>>
>>> runltp has been removed from ltp and kirk is the official tool to
>>> invoke linux ltp tests.
>>>
>>> See:
>>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
>>> ub.com%2Flinux-test-
>> project%2Fltp%2Fcommit%2F6efd3605dc005c3ed135b463f
>>>
>> 182174e24bdce1b&data=05%7C02%7Cdaniel.turull%40ericsson.com%7Cca4e75
>> 36
>>>
>> 591a44feb4fd08de8a8c9c1f%7C92e84cebfbfd47abbe52080c6b87953f%7C0%7C
>> 0%7C
>>>
>> 639100533691606542%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy
>> dWUsIlY
>>>
>> iOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C
>> 0%
>>>
>> 7C%7C%7C&sdata=cgqr%2FrR99Ty%2F75pasLCYawpw0qV%2FAZ4sXUDWH%2B2
>> ka9c%3D&
>>> reserved=0
>>>
>>> Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
>>> Assisted-by: Claude, Anthropic
>>> ---
>>>  meta/lib/oeqa/runtime/cases/ltp.py        | 16 ++++++-------
>>>  meta/lib/oeqa/runtime/cases/ltp_stress.py | 16 +++++++++----
>>>  meta/lib/oeqa/utils/logparser.py          | 29
>>> +++++++++++++++++++++++
>>>  meta/recipes-extended/ltp/ltp_20260130.bb |  1 +
>>>  4 files changed, 50 insertions(+), 12 deletions(-)
>>>
>>> diff --git a/meta/lib/oeqa/runtime/cases/ltp.py
>>> b/meta/lib/oeqa/runtime/cases/ltp.py
>>> index 0ffdbe23e4..11c4814090 100644
>>> --- a/meta/lib/oeqa/runtime/cases/ltp.py
>>> +++ b/meta/lib/oeqa/runtime/cases/ltp.py
>>> @@ -12,7 +12,7 @@ import pprint
>>>  from oeqa.runtime.case import OERuntimeTestCase
>>>  from oeqa.core.decorator.depends import OETestDepends
>>>  from oeqa.runtime.decorator.package import OEHasPackage -from
>>> oeqa.utils.logparser import LtpParser
>>> +from oeqa.utils.logparser import LtpKirkParser
>>>
>>>  class LtpTestBase(OERuntimeTestCase):
>>>
>>> @@ -66,9 +66,9 @@ class LtpTest(LtpTestBase):
>>>
>>>      def runltp(self, ltp_group):
>>>              # LTP appends to log files, so ensure we start with a
>>> clean log
>>> -            self.target.deleteFiles("/opt/ltp/results/", ltp_group)
>>> +            self.target.deleteFiles("/opt/ltp/results/", "%s.json" %
>>> +ltp_group)
>>>
>>> -            cmd = '/opt/ltp/runltp -f %s -q -r /opt/ltp -l
>>> /opt/ltp/results/%s -I 1 -d /opt/ltp' % (ltp_group, ltp_group)
>>> +            cmd = 'kirk --run-suite %s --json-report
>>> +/opt/ltp/results/%s.json -n' % (ltp_group, ltp_group)
>>>
>>>              starttime = time.time()
>>>              (status, output) = self.target.run(cmd, timeout=1200) @@
>>> -87,14 +87,14 @@ class LtpTest(LtpTestBase):
>>>              self.extras['ltpresult.rawlogs']['log'] =
>>> self.extras['ltpresult.rawlogs']['log'] + output
>>>
>>>              # Copy the machine-readable test results locally so we
>>> can parse it
>>> -            dst = os.path.join(self.ltptest_log_dir, ltp_group)
>>> -            remote_src = "/opt/ltp/results/%s" % ltp_group
>>> +            dst = os.path.join(self.ltptest_log_dir, "%s.json" %
>>> +ltp_group)
>>> +            remote_src = "/opt/ltp/results/%s.json" % ltp_group
>>>              (status, output) = self.target.copyFrom(remote_src, dst,
>>> True)
>>>              if status:
>>>                  msg = 'File could not be copied. Output: %s' % output
>>>                  self.target.logger.warning(msg)
>>>
>>> -            parser = LtpParser()
>>> +            parser = LtpKirkParser()
>>>              results, sections  = parser.parse(dst)
>>>
>>>              sections['duration'] = int(endtime-starttime) @@ -113,9
>>> +113,9 @@ class LtpTest(LtpTestBase):
>>>
>>>      # LTP runtime tests
>>>      @OETestDepends(['ssh.SSHTest.test_ssh'])
>>> -    @OEHasPackage(["ltp"])
>>> +    @OEHasPackage(["ltp", "python3-kirk"])
>>>      def test_ltp_help(self):
>>> -        (status, output) = self.target.run('/opt/ltp/runltp --help')
>>> +        (status, output) = self.target.run('kirk --help')
>>>          msg = 'Failed to get ltp help. Output: %s' % output
>>>          self.assertEqual(status, 0, msg=msg)
>>>
>>> diff --git a/meta/lib/oeqa/runtime/cases/ltp_stress.py
>>> b/meta/lib/oeqa/runtime/cases/ltp_stress.py
>>> index ce6f4bf59d..cf84ec1182 100644
>>> --- a/meta/lib/oeqa/runtime/cases/ltp_stress.py
>>> +++ b/meta/lib/oeqa/runtime/cases/ltp_stress.py
>>> @@ -13,7 +13,7 @@ from oeqa.runtime.case import OERuntimeTestCase
>>>  from oeqa.core.decorator.depends import OETestDepends
>>>  from oeqa.runtime.decorator.package import OEHasPackage
>>>  from oeqa.core.decorator.data import skipIfQemu -from
>>> oeqa.utils.logparser import LtpParser
>>> +from oeqa.utils.logparser import LtpKirkParser
>>>
>>>  class LtpStressBase(OERuntimeTestCase):
>>>
>>> @@ -60,7 +60,7 @@ class LtpStressBase(OERuntimeTestCase):
>>>  class LtpStressTest(LtpStressBase):
>>>
>>>      def runltp(self, stress_group):
>>> -            cmd = '/opt/ltp/runltp -f %s -p -q 2>@1 | tee
>>> /opt/ltp/results/%s' % (stress_group, stress_group)
>>> +            cmd = 'kirk --run-suite %s --json-report
>>> +/opt/ltp/results/%s.json -n' % (stress_group, stress_group)
>>>              starttime = time.time()
>>>              (status, output) = self.target.run(cmd)
>>>              endtime = time.time()
>>> @@ -69,8 +69,16 @@ class LtpStressTest(LtpStressBase):
>>>
>>>              self.extras['ltpstressresult.rawlogs']['log'] =
>>> self.extras['ltpstressresult.rawlogs']['log'] + output
>>>
>>> -            parser = LtpParser()
>>> -            results, sections  =
>>> parser.parse(os.path.join(self.ltptest_log_dir, "%s" % stress_group))
>>> +            # Copy kirk JSON report from target
>>> +            dst = os.path.join(self.ltptest_log_dir, "%s.json" %
>>> +stress_group)
>>> +            remote_src = "/opt/ltp/results/%s.json" % stress_group
>>> +            (status, output) = self.target.copyFrom(remote_src, dst,
>>> +True)
>>> +            if status:
>>> +                msg = 'File could not be copied. Output: %s' % output
>>> +                self.target.logger.warning(msg)
>>> +
>>> +            parser = LtpKirkParser()
>>> +            results, sections  = parser.parse(dst)
>>>
>>>              runtime = int(endtime-starttime)
>>>              sections['duration'] = runtime diff --git
>>> a/meta/lib/oeqa/utils/logparser.py b/meta/lib/oeqa/utils/logparser.py
>>> index c479864162..a907421fab 100644
>>> --- a/meta/lib/oeqa/utils/logparser.py
>>> +++ b/meta/lib/oeqa/utils/logparser.py
>>> @@ -5,6 +5,7 @@
>>>  #
>>>
>>>  import enum
>>> +import json
>>>  import os
>>>  import re
>>>
>>> @@ -158,6 +159,34 @@ class LtpParser:
>>>          return results, section
>>>
>>>
>>> +class LtpKirkParser:
>>> +    """Parse kirk JSON report into the same format as LtpParser."""
>>> +
>>> +    STATUS_MAP = {
>>> +        "pass": "PASSED",
>>> +        "fail": "FAILED",
>>> +        "brok": "FAILED",
>>> +        "conf": "SKIPPED",
>>> +        "warn": "PASSED",
>>> +    }
>>> +
>>> +    def parse(self, jsonfile):
>>> +        with open(jsonfile, errors="replace") as f:
>>> +            report = json.load(f)
>>> +
>>> +        results = {}
>>> +        section = {"duration": 0, "log": ""}
>>> +
>>> +        for entry in report.get("results", []):
>>> +            results[entry["test_fqn"]] =
>>> +self.STATUS_MAP.get(entry.get("status", ""), "FAILED")
>>> +            test = entry.get("test", {})
>>> +            section["log"] += test.get("log", "")
>>> +
>>> +        section["duration"] = int(report.get("stats",
>>> +{}).get("runtime", 0))
>>> +
>>> +        return results, section
>>> +
>>> +
>>
>> FWIW we can probably just update the LtpParser class directly rather than
>> creating a wrapper around it since runltp will be no more in the next release
>> anyway.
>>
>> Cheers,
>>
>> Richard
>