[PATCH 1/1] ltp: upgrade 20250530 -> 20250930

[PATCH 1/1] ltp: upgrade 20250530 -> 20250930

[Petr Vorel]

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
---
Hi all,

I'm sorry, I planned to test this time, but it failed. Any hint what's
wrong? Or can anybody test this patch?

$ . oe-init-build-env

.../build $ bitbake ltp
ERROR: Error importing OE modules: module 'bb.parse' has no attribute
'vardepsexclude'
ERROR: Unable to parse
/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py
Traceback (most recent call last):
  File
  "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py",
  line 372, in eval
      layerid, fragment_name = f.split('/', 1)
	      ^^^^^^^^^^^^^^^^^^^^^^
		  ValueError: not enough values to unpack (expected 2, got 1)

FYI I also plan to get rid of some patches posted.

* 0001-Add-__clear_cache-declaration-for-clang.patch
It's wrong, I'd like to replace it with:
https://patchwork.ozlabs.org/project/ltp/list/?series=477319&state=*

* 0001-syscalls-semctl08-Skip-semctl08-when-__USE_TIME64_RE.patch
I posted a review, hopefully we get a feedback from other developers
soon.

* 0001-Remove-OOM-tests-from-runtest-mm.patch
Is it really needed to drop these tests?

BTW for those who run LTP: kirk might be interesting tool to use:
https://github.com/linux-test-project/kirk

We also bundle it in LTP as a submodule:
https://github.com/linux-test-project/ltp/blob/master/.gitmodules

It's also in pypi:
https://pypi.org/project/kirk/

I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated
/opt/ltp/runltp is still being used. We want to remove it (not sure
when, but it will happen sooner or later). Any change somebody would
submit a patch to switch to kirk?

Kind regards,
Petr

 .../0001-Remove-OOM-tests-from-runtest-mm.patch    | 13 ++++++++-----
 ...mctl08-Skip-semctl08-when-__USE_TIME64_RE.patch | 14 +++++++-------
 .../ltp/{ltp_20250530.bb => ltp_20250930.bb}       |  2 +-
 3 files changed, 16 insertions(+), 13 deletions(-)
 rename meta/recipes-extended/ltp/{ltp_20250530.bb => ltp_20250930.bb} (99%)

diff --git a/meta/recipes-extended/ltp/ltp/0001-Remove-OOM-tests-from-runtest-mm.patch b/meta/recipes-extended/ltp/ltp/0001-Remove-OOM-tests-from-runtest-mm.patch
index 936e23ebda..c1cf8e8a41 100644
--- a/meta/recipes-extended/ltp/ltp/0001-Remove-OOM-tests-from-runtest-mm.patch
+++ b/meta/recipes-extended/ltp/ltp/0001-Remove-OOM-tests-from-runtest-mm.patch
@@ -1,23 +1,23 @@
-From 7096737fbbe19d0765f0a8c62ef7667bf4875780 Mon Sep 17 00:00:00 2001
+From 3e59500d1dc60686b8c4c6a24813bd14ad479e43 Mon Sep 17 00:00:00 2001
 From: "Mingde (Matthew) Zeng" <matthewzmd@gmail.com>
 Date: Wed, 29 Jul 2020 08:47:09 -0400
-Subject: [PATCH] Remove OOM tests from runtest/mm
+Subject: [PATCH 1/3] Remove OOM tests from runtest/mm
 
 Disable OOM tests, as they might cause oeqa ssh connection lost
 
 Upstream-Status: Inappropriate [oe-core specific]
 Signed-off-by: Mingde (Matthew) Zeng <matthew.zeng@windriver.com>
-[ pvorel: rebased for 20210927 ]
+[ pvorel: rebased for 20250930 ]
 Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
 ---
  runtest/mm | 6 ------
  1 file changed, 6 deletions(-)
 
 diff --git a/runtest/mm b/runtest/mm
-index 5566a7742..8014d509b 100644
+index 41d624ad86..1e8c226389 100644
 --- a/runtest/mm
 +++ b/runtest/mm
-@@ -70,12 +70,6 @@ ksm07 ksm07
+@@ -69,12 +69,6 @@ ksm07 ksm07
  cpuset01 cpuset01
  cpuset02 cpuset02
  
@@ -30,3 +30,6 @@ index 5566a7742..8014d509b 100644
  swapping01 swapping01 -i 5
  
  thp01 thp01 -I 120
+-- 
+2.51.0
+
diff --git a/meta/recipes-extended/ltp/ltp/0001-syscalls-semctl08-Skip-semctl08-when-__USE_TIME64_RE.patch b/meta/recipes-extended/ltp/ltp/0001-syscalls-semctl08-Skip-semctl08-when-__USE_TIME64_RE.patch
index b4859a6f0a..0c7a351b7e 100644
--- a/meta/recipes-extended/ltp/ltp/0001-syscalls-semctl08-Skip-semctl08-when-__USE_TIME64_RE.patch
+++ b/meta/recipes-extended/ltp/ltp/0001-syscalls-semctl08-Skip-semctl08-when-__USE_TIME64_RE.patch
@@ -1,8 +1,8 @@
-From 55b48d66857a43c2609fc351293b5601e2eb955d Mon Sep 17 00:00:00 2001
+From 1285a11d8f7660dc42afb5f3fcf147d54e321844 Mon Sep 17 00:00:00 2001
 From: Jiaying Song <jiaying.song.cn@windriver.com>
 Date: Fri, 23 May 2025 15:17:49 +0800
-Subject: [PATCH] syscalls/semctl08: Skip semctl08 when __USE_TIME64_REDIRECTS
- is defined
+Subject: [PATCH 3/3] syscalls/semctl08: Skip semctl08 when
+ __USE_TIME64_REDIRECTS is defined
 
 When __USE_TIME64_REDIRECTS is defined, glibc redirects struct semid_ds to a
 64-bit time-safe version that omits the sem_otime_high and sem_ctime_high
@@ -20,10 +20,10 @@ Signed-off-by: Jiaying Song <jiaying.song.cn@windriver.com>
  1 file changed, 5 insertions(+), 3 deletions(-)
 
 diff --git a/testcases/kernel/syscalls/ipc/semctl/semctl08.c b/testcases/kernel/syscalls/ipc/semctl/semctl08.c
-index 1878bd4..3b799fa 100644
+index f4549adf43..28776f266e 100644
 --- a/testcases/kernel/syscalls/ipc/semctl/semctl08.c
 +++ b/testcases/kernel/syscalls/ipc/semctl/semctl08.c
-@@ -10,7 +10,11 @@
+@@ -12,7 +12,11 @@
  #include "tst_test.h"
  #include "libnewipc.h"
  
@@ -36,7 +36,7 @@ index 1878bd4..3b799fa 100644
  
  static void run(void)
  {
-@@ -47,6 +51,4 @@ static struct tst_test test = {
+@@ -49,6 +53,4 @@ static struct tst_test test = {
  	.test_all = run,
  	.needs_tmpdir = 1,
  };
@@ -44,5 +44,5 @@ index 1878bd4..3b799fa 100644
 -TST_TEST_TCONF("test requires struct semid64_ds to have the time_high fields");
  #endif
 -- 
-2.34.1
+2.51.0
 
diff --git a/meta/recipes-extended/ltp/ltp_20250530.bb b/meta/recipes-extended/ltp/ltp_20250930.bb
similarity index 99%
rename from meta/recipes-extended/ltp/ltp_20250530.bb
rename to meta/recipes-extended/ltp/ltp_20250930.bb
index 9ea5de10ee..1514aca8fe 100644
--- a/meta/recipes-extended/ltp/ltp_20250530.bb
+++ b/meta/recipes-extended/ltp/ltp_20250930.bb
@@ -24,7 +24,7 @@ TUNE_CCARGS:remove:x86-64 = "-mfpmath=sse"
 
 CFLAGS:append:powerpc64 = " -D__SANE_USERSPACE_TYPES__"
 CFLAGS:append:mipsarchn64 = " -D__SANE_USERSPACE_TYPES__"
-SRCREV = "14331e1ecfcd63426c9d270d88b7bad9f60c6d64"
+SRCREV = "d2550ffbbcfe163212cd7e9c132db65ae0fa06ed"
 
 SRC_URI = "git://github.com/linux-test-project/ltp.git;branch=master;protocol=https \
            file://0001-Remove-OOM-tests-from-runtest-mm.patch \
-- 
2.51.0

Re: [PATCH 1/1] ltp: upgrade 20250530 -> 20250930

[Richard Purdie]

On Sun, 2025-10-12 at 22:08 +0200, Petr Vorel wrote:
> Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
> ---
> Hi all,
> 
> I'm sorry, I planned to test this time, but it failed. Any hint what's
> wrong? Or can anybody test this patch?
> 
> $ . oe-init-build-env
> 
> .../build $ bitbake ltp
> ERROR: Error importing OE modules: module 'bb.parse' has no attribute
> 'vardepsexclude'
> ERROR: Unable to parse
> /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py
> Traceback (most recent call last):
>   File
>   "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py",
>   line 372, in eval
>       layerid, fragment_name = f.split('/', 1)
> 	      ^^^^^^^^^^^^^^^^^^^^^^
> 		  ValueError: not enough values to unpack (expected 2, got 1)
> 
> FYI I also plan to get rid of some patches posted.

Are you setting OE_FRAGMENTS to something in a config file? It
shouldn't traceback like this so that is a bug but something is
triggering it...

> * 0001-Add-__clear_cache-declaration-for-clang.patch
> It's wrong, I'd like to replace it with:
> https://patchwork.ozlabs.org/project/ltp/list/?series=477319&state=*
> 
> * 0001-syscalls-semctl08-Skip-semctl08-when-__USE_TIME64_RE.patch
> I posted a review, hopefully we get a feedback from other developers
> soon.
> 
> * 0001-Remove-OOM-tests-from-runtest-mm.patch
> Is it really needed to drop these tests?
> 
> BTW for those who run LTP: kirk might be interesting tool to use:
> https://github.com/linux-test-project/kirk

I did send some questions and had some discussion on kirk a while ago.
Quite simply, it isn't useful/interesting to Yocto Project.

What we want to test with is our images and our kernel, as we build it.
kirk, as far as I understand it has gone a different route where there
isn't really any userspace left and it simply tests against a kernel
binary. We'd no longer be testing our build artefacts but some more
artificial construct.

We're trying to test what we build. You're trying to test the kernel
for regressions. They're two different things.

I totally understand why you've gone that direction with kirk but I
also did spell out at the time that it wasn't something which really
fits in with the way we run tests, or what we actually aim to test. I
was told at the time that basically, nobody is interested in what we
want/do.

> We also bundle it in LTP as a submodule:
> https://github.com/linux-test-project/ltp/blob/master/.gitmodules
> 
> It's also in pypi:
> https://pypi.org/project/kirk/
> 
> I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated
> /opt/ltp/runltp is still being used. We want to remove it (not sure
> when, but it will happen sooner or later). Any change somebody would
> submit a patch to switch to kirk?

It is more likely that when you drop runltp, we'll just have to drop
ltp. Sorry :(. I did explain this at while ago :/.

I did also flag the issue to our own wider community but nobody has
stepped up to do anything about it. I've way too many other things to
work on so it is very unlikely I have any time to look at it.

Cheers,

Richard

Re: [PATCH 1/1] ltp: upgrade 20250530 -> 20250930

[Petr Vorel]

Hi Richard, all,

[ I accidentally deleted mail thread. Unfortunately I don't see Message-id in
web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other
LTP developers about runltp vs. kirk would not be in the thread ]

> > $ . oe-init-build-env
> > 
> > .../build $ bitbake ltp
> > ERROR: Error importing OE modules: module 'bb.parse' has no attribute
> > 'vardepsexclude'
> > ERROR: Unable to parse
> > /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py
> > Traceback (most recent call last):
> >   File
> >   "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py",
> >   line 372, in eval
> >       layerid, fragment_name = f.split('/', 1)
> > 	      ^^^^^^^^^^^^^^^^^^^^^^
> > 		  ValueError: not enough values to unpack (expected 2, got 1)
> > 

> > FYI I also plan to get rid of some patches posted.

> Are you setting OE_FRAGMENTS to something in a config file? It
> shouldn't traceback like this so that is a bug but something is
> triggering it...

No, simple clone and the 2 commands above. I'll later try it again on fresh
git clone.

> I did send some questions and had some discussion on kirk a while ago.
> Quite simply, it isn't useful/interesting to Yocto Project.

I see your points [2] [3].

> What we want to test with is our images and our kernel, as we build it.
> kirk, as far as I understand it has gone a different route where there
> isn't really any userspace left and it simply tests against a kernel
> binary. We'd no longer be testing our build artefacts but some more
> artificial construct.

I don't understand "any userspace left and it simply tests against a kernel
binary". LTP tests are mostly focused on the kernel (+ it's modules). And you
can run individual tests by just executing them (+ handle environment variables)
or use runltp or use kirk. The executor does not matter that much. In the end we
at SUSE also test with LTP our built image :). (LTP is used by mainline folks
and by distro folks).

FYI although kirk is meant to be run on the host, doing a different connections,
it can also be run on SUT. Sure, there is then python3 dependency on SUT
(heavier than shell + it's dependencies), but still kind of runltp replacement.

> 
> We're trying to test what we build. You're trying to test the kernel
> for regressions. They're two different things.
> 
> I totally understand why you've gone that direction with kirk but I
> also did spell out at the time that it wasn't something which really
> fits in with the way we run tests, or what we actually aim to test. I
> was told at the time that basically, nobody is interested in what we
> want/do.

...
> > I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated
> > /opt/ltp/runltp is still being used. We want to remove it (not sure
> > when, but it will happen sooner or later). Any change somebody would
> > submit a patch to switch to kirk?

> It is more likely that when you drop runltp, we'll just have to drop
> ltp. Sorry :(. I did explain this at while ago :/.

It's a question if any of the users really need LTP. If yes, you could vendor
runltp.

Or, write a simple script which parses the content of the runtest files and
execute them. FYI for part of openSUSE testing we still use our custom openQA
runner which does exactly this. It would be very light: -d and -r can be
replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing
thing will be generating of results (if you really need it, I'd recommend kirk
and it's JSON output).

The reason we go to kirk + ltx is that in the future we plan to get rid of
runtests, replace them with metadata info (that will allow many things [4],
but you were Cc at the discussion [5]). Once this happen, runltp (or any custom
script / framework runner) will be broken. But this likely take long time.

> I did also flag the issue to our own wider community but nobody has
> stepped up to do anything about it. I've way too many other things to
> work on so it is very unlikely I have any time to look at it.

Understand, everybody is busy with the 

Kind regards,
Petr

[1] https://lists.openembedded.org/g/openembedded-core/message/224749
[2] https://lore.kernel.org/ltp/c8d4ee181809c4bbf5e21bf355c241eeb540e9a5.camel@linuxfoundation.org/
[3] https://lore.kernel.org/ltp/8043628a6eed94e788f9fedbf6c8b264ebfbae15.camel@linuxfoundation.org/
[4] https://www.youtube.com/watch?v=T1ImuNr9Oxo
[5] https://lore.kernel.org/ltp/ZmbFyjuXndeXCLp8@rei/

Re: [PATCH 1/1] ltp: upgrade 20250530 -> 20250930

[Richard Purdie]

On Mon, 2025-10-13 at 20:59 +0200, Petr Vorel wrote:
> Hi Richard, all,
> 
> [ I accidentally deleted mail thread. Unfortunately I don't see Message-id in
> web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other
> LTP developers about runltp vs. kirk would not be in the thread ]
> 
> > > $ . oe-init-build-env
> > > 
> > > .../build $ bitbake ltp
> > > ERROR: Error importing OE modules: module 'bb.parse' has no attribute
> > > 'vardepsexclude'
> > > ERROR: Unable to parse
> > > /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py
> > > Traceback (most recent call last):
> > >   File
> > >   "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py",
> > >   line 372, in eval
> > >       layerid, fragment_name = f.split('/', 1)
> > > 	      ^^^^^^^^^^^^^^^^^^^^^^
> > > 		  ValueError: not enough values to unpack (expected 2, got 1)
> > > 
> 
> > > FYI I also plan to get rid of some patches posted.
> 
> > Are you setting OE_FRAGMENTS to something in a config file? It
> > shouldn't traceback like this so that is a bug but something is
> > triggering it...
> 
> No, simple clone and the 2 commands above. I'll later try it again on fresh
> git clone.

If you could share the repo/branch you cloned (poky?) I'd appreciate it
as I did try and reproduce that and couldn't. I will write some patches
to improve the code/error regardless as that much is clear from the
failure. It really shouldn't fail like that, it is embarrassing :/.

> > I did send some questions and had some discussion on kirk a while ago.
> > Quite simply, it isn't useful/interesting to Yocto Project.
> 
> I see your points [2] [3].
> 
> > What we want to test with is our images and our kernel, as we build it.
> > kirk, as far as I understand it has gone a different route where there
> > isn't really any userspace left and it simply tests against a kernel
> > binary. We'd no longer be testing our build artefacts but some more
> > artificial construct.
> 
> I don't understand "any userspace left and it simply tests against a kernel
> binary". LTP tests are mostly focused on the kernel (+ it's modules). And you
> can run individual tests by just executing them (+ handle environment variables)
> or use runltp or use kirk. The executor does not matter that much. In the end we
> at SUSE also test with LTP our built image :). (LTP is used by mainline folks
> and by distro folks).

Sorry, I'm not really being clear. The Yocto Project is in the middle
of some huge changes and I'm struggling with those. Equally, I did want
to at least given some response to your upgrade which is appreciated.

I guess we're one of the few ltp users with a cross environment. We
have a way to launch target images under emulation and our own methods
for controlling them and transferring files. We do all this without
special permissions or access other than ability to use kvm. As such we
liked being able to just run runltp on the target in the environment we
already have.

kirk, at least as I understand it, wants to do much of this for itself.
That means we end up with two ways of running the target emulation
which may or may not match. We'd much prefer just having one so we
either have issues there or we do not but our test environment is
consistent across all tests.

> FYI although kirk is meant to be run on the host, doing a different connections,
> it can also be run on SUT. Sure, there is then python3 dependency on SUT
> (heavier than shell + it's dependencies), but still kind of runltp replacement.
> 
> > 
> > We're trying to test what we build. You're trying to test the kernel
> > for regressions. They're two different things.
> > 
> > I totally understand why you've gone that direction with kirk but I
> > also did spell out at the time that it wasn't something which really
> > fits in with the way we run tests, or what we actually aim to test. I
> > was told at the time that basically, nobody is interested in what we
> > want/do.
> 
> ...
> > > I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated
> > > /opt/ltp/runltp is still being used. We want to remove it (not sure
> > > when, but it will happen sooner or later). Any change somebody would
> > > submit a patch to switch to kirk?
> 
> > It is more likely that when you drop runltp, we'll just have to drop
> > ltp. Sorry :(. I did explain this at while ago :/.
> 
> It's a question if any of the users really need LTP. If yes, you could vendor
> runltp.
> 
> Or, write a simple script which parses the content of the runtest files and
> execute them. FYI for part of openSUSE testing we still use our custom openQA
> runner which does exactly this. It would be very light: -d and -r can be
> replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing
> thing will be generating of results (if you really need it, I'd recommend kirk
> and it's JSON output).

We'd appreciate and be able to use the json output, we already have
json output for most of our other tests. Beyond that, as you say, we
don't really need much beyond what runltp does though. kirk brings with
it a number of things we definitely do not want (as mentioned above). I
don't know if we can avoid those or not.

> The reason we go to kirk + ltx is that in the future we plan to get rid of
> runtests, replace them with metadata info (that will allow many things [4],
> but you were Cc at the discussion [5]). Once this happen, runltp (or any custom
> script / framework runner) will be broken. But this likely take long time.

Right, using a vendored runltp would just buy us a small amount of time
so likely isn't a long term viable solution sadly. I'd consider it if
it were but it doesn't seem a good investment of time.

Cheers,

Richard

Re: [PATCH 1/1] ltp: upgrade 20250530 -> 20250930

[Petr Vorel]

Hi Richard,

> On Mon, 2025-10-13 at 20:59 +0200, Petr Vorel wrote:
> > Hi Richard, all,

> > [ I accidentally deleted mail thread. Unfortunately I don't see Message-id in
> > web UI [1] therefore I cannot set In-Reply-To:. Due this also any reply to other
> > LTP developers about runltp vs. kirk would not be in the thread ]

> > > > $ . oe-init-build-env

> > > > .../build $ bitbake ltp
> > > > ERROR: Error importing OE modules: module 'bb.parse' has no attribute
> > > > 'vardepsexclude'
> > > > ERROR: Unable to parse
> > > > /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py
> > > > Traceback (most recent call last):
> > > > ďż˝ File
> > > > ďż˝ "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py",
> > > > ďż˝ line 372, in eval
> > > > ����� layerid, fragment_name = f.split('/', 1)
> > > > 	����� ^^^^^^^^^^^^^^^^^^^^^^
> > > > 		ďż˝ ValueError: not enough values to unpack (expected 2, got 1)


> > > > FYI I also plan to get rid of some patches posted.

> > > Are you setting OE_FRAGMENTS to something in a config file? It
> > > shouldn't traceback like this so that is a bug but something is
> > > triggering it...

> > No, simple clone and the 2 commands above. I'll later try it again on fresh
> > git clone.

> If you could share the repo/branch you cloned (poky?) I'd appreciate it
> as I did try and reproduce that and couldn't. I will write some patches
> to improve the code/error regardless as that much is clear from the
> failure. It really shouldn't fail like that, it is embarrassing :/.

Master branch from my fork [1] do:

... $ . oe-init-build-env
./build $ bitbake ltp

ERROR: Error importing OE modules: module 'bb.parse' has no attribute 'vardepsexclude'
ERROR: Unable to parse /home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py
Traceback (most recent call last):
  File "/home/pvorel/install/src/openembedded/bitbake/lib/bb/parse/ast.py", line 372, in eval
    layerid, fragment_name = f.split('/', 1)
    ^^^^^^^^^^^^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

Kind regards,
Petr

[1] https://github.com/pevik/openembedded-core

Re: [PATCH 1/1] ltp: upgrade 20250530 -> 20250930

[Petr Vorel]

Hi Richard, all,

[ I realized that we went quite far to kirk topic, which would be interesting to
Cyril, Andrea and possible other people developers or users, therefore Cc them.
Replying to this thread:
https://lists.openembedded.org/g/openembedded-core/message/224816
+ links to replies from Richardo previously in LTP ML
https://lore.kernel.org/ltp/c8d4ee181809c4bbf5e21bf355c241eeb540e9a5.camel@linuxfoundation.org/
https://lore.kernel.org/ltp/8043628a6eed94e788f9fedbf6c8b264ebfbae15.camel@linuxfoundation.org/
NOTE: Cc requires subscription.
]

I added few final points below.

> On Mon, 2025-10-13 at 20:59 +0200, Petr Vorel wrote:

...
> > > I did send some questions and had some discussion on kirk a while ago.
> > > Quite simply, it isn't useful/interesting to Yocto Project.

> > I see your points [2] [3].

> > > What we want to test with is our images and our kernel, as we build it.
> > > kirk, as far as I understand it has gone a different route where there
> > > isn't really any userspace left and it simply tests against a kernel
> > > binary. We'd no longer be testing our build artefacts but some more
> > > artificial construct.

> > I don't understand "any userspace left and it simply tests against a kernel
> > binary". LTP tests are mostly focused on the kernel (+ it's modules). And you
> > can run individual tests by just executing them (+ handle environment variables)
> > or use runltp or use kirk. The executor does not matter that much. In the end we
> > at SUSE also test with LTP our built image :). (LTP is used by mainline folks
> > and by distro folks).

> Sorry, I'm not really being clear. The Yocto Project is in the middle
> of some huge changes and I'm struggling with those. Equally, I did want
> to at least given some response to your upgrade which is appreciated.

> I guess we're one of the few ltp users with a cross environment. We
> have a way to launch target images under emulation and our own methods
> for controlling them and transferring files. We do all this without
> special permissions or access other than ability to use kvm. As such we
> liked being able to just run runltp on the target in the environment we
> already have.

> kirk, at least as I understand it, wants to do much of this for itself.
> That means we end up with two ways of running the target emulation
> which may or may not match. We'd much prefer just having one so we
> either have issues there or we do not but our test environment is
> consistent across all tests.

> > FYI although kirk is meant to be run on the host, doing a different connections,
> > it can also be run on SUT. Sure, there is then python3 dependency on SUT
> > (heavier than shell + it's dependencies), but still kind of runltp replacement.


> > > We're trying to test what we build. You're trying to test the kernel
> > > for regressions. They're two different things.

> > > I totally understand why you've gone that direction with kirk but I
> > > also did spell out at the time that it wasn't something which really
> > > fits in with the way we run tests, or what we actually aim to test. I
> > > was told at the time that basically, nobody is interested in what we
> > > want/do.

> > ...
> > > > I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated
> > > > /opt/ltp/runltp is still being used. We want to remove it (not sure
> > > > when, but it will happen sooner or later). Any change somebody would
> > > > submit a patch to switch to kirk?

> > > It is more likely that when you drop runltp, we'll just have to drop
> > > ltp. Sorry :(. I did explain this at while ago :/.

> > It's a question if any of the users really need LTP. If yes, you could vendor
> > runltp.

> > Or, write a simple script which parses the content of the runtest files and
> > execute them. FYI for part of openSUSE testing we still use our custom openQA
> > runner which does exactly this. It would be very light: -d and -r can be
> > replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing
> > thing will be generating of results (if you really need it, I'd recommend kirk
> > and it's JSON output).

> We'd appreciate and be able to use the json output, we already have
> json output for most of our other tests. Beyond that, as you say, we
> don't really need much beyond what runltp does though. kirk brings with
> it a number of things we definitely do not want (as mentioned above). I
> don't know if we can avoid those or not.

As I wrote previously, kirk is possible to be used as nearly drop-in runltp
replacement run on the SUT. It's not the intended use-case (we aim it to be run
on host), but still supported use case.

> > The reason we go to kirk + ltx is that in the future we plan to get rid of
> > runtests, replace them with metadata info (that will allow many things [4],
> > but you were Cc at the discussion [5]). Once this happen, runltp (or any custom
> > script / framework runner) will be broken. But this likely take long time.

> Right, using a vendored runltp would just buy us a small amount of time
> so likely isn't a long term viable solution sadly. I'd consider it if
> it were but it doesn't seem a good investment of time.

I would say adopting kirk in minimalistic way (or identify what you need and
it's still missing) would be sure safer in a long term.

Kind regards,
Petr

> Cheers,

> Richard