Re: [LTP] [PATCH 1/1] ltp: upgrade 20250530 -> 20250930

[Petr Vorel <petr.vorel@gmail.com>]

Hi Richard, all,

[ I realized that we went quite far to kirk topic, which would be interesting to
Cyril, Andrea and possible other people developers or users, therefore Cc them.
Replying to this thread:
https://lists.openembedded.org/g/openembedded-core/message/224816
+ links to replies from Richardo previously in LTP ML
https://lore.kernel.org/ltp/c8d4ee181809c4bbf5e21bf355c241eeb540e9a5.camel@linuxfoundation.org/
https://lore.kernel.org/ltp/8043628a6eed94e788f9fedbf6c8b264ebfbae15.camel@linuxfoundation.org/
NOTE: Cc requires subscription.
]

I added few final points below.

> On Mon, 2025-10-13 at 20:59 +0200, Petr Vorel wrote:

...
> > > I did send some questions and had some discussion on kirk a while ago.
> > > Quite simply, it isn't useful/interesting to Yocto Project.

> > I see your points [2] [3].

> > > What we want to test with is our images and our kernel, as we build it.
> > > kirk, as far as I understand it has gone a different route where there
> > > isn't really any userspace left and it simply tests against a kernel
> > > binary. We'd no longer be testing our build artefacts but some more
> > > artificial construct.

> > I don't understand "any userspace left and it simply tests against a kernel
> > binary". LTP tests are mostly focused on the kernel (+ it's modules). And you
> > can run individual tests by just executing them (+ handle environment variables)
> > or use runltp or use kirk. The executor does not matter that much. In the end we
> > at SUSE also test with LTP our built image :). (LTP is used by mainline folks
> > and by distro folks).

> Sorry, I'm not really being clear. The Yocto Project is in the middle
> of some huge changes and I'm struggling with those. Equally, I did want
> to at least given some response to your upgrade which is appreciated.

> I guess we're one of the few ltp users with a cross environment. We
> have a way to launch target images under emulation and our own methods
> for controlling them and transferring files. We do all this without
> special permissions or access other than ability to use kvm. As such we
> liked being able to just run runltp on the target in the environment we
> already have.

> kirk, at least as I understand it, wants to do much of this for itself.
> That means we end up with two ways of running the target emulation
> which may or may not match. We'd much prefer just having one so we
> either have issues there or we do not but our test environment is
> consistent across all tests.

> > FYI although kirk is meant to be run on the host, doing a different connections,
> > it can also be run on SUT. Sure, there is then python3 dependency on SUT
> > (heavier than shell + it's dependencies), but still kind of runltp replacement.


> > > We're trying to test what we build. You're trying to test the kernel
> > > for regressions. They're two different things.

> > > I totally understand why you've gone that direction with kirk but I
> > > also did spell out at the time that it wasn't something which really
> > > fits in with the way we run tests, or what we actually aim to test. I
> > > was told at the time that basically, nobody is interested in what we
> > > want/do.

> > ...
> > > > I see in meta/lib/oeqa/runtime/cases/ltp.py the deprecated
> > > > /opt/ltp/runltp is still being used. We want to remove it (not sure
> > > > when, but it will happen sooner or later). Any change somebody would
> > > > submit a patch to switch to kirk?

> > > It is more likely that when you drop runltp, we'll just have to drop
> > > ltp. Sorry :(. I did explain this at while ago :/.

> > It's a question if any of the users really need LTP. If yes, you could vendor
> > runltp.

> > Or, write a simple script which parses the content of the runtest files and
> > execute them. FYI for part of openSUSE testing we still use our custom openQA
> > runner which does exactly this. It would be very light: -d and -r can be
> > replaced by TMPDIR and LTPROOT, -I is supported by all tests. The only missing
> > thing will be generating of results (if you really need it, I'd recommend kirk
> > and it's JSON output).

> We'd appreciate and be able to use the json output, we already have
> json output for most of our other tests. Beyond that, as you say, we
> don't really need much beyond what runltp does though. kirk brings with
> it a number of things we definitely do not want (as mentioned above). I
> don't know if we can avoid those or not.

As I wrote previously, kirk is possible to be used as nearly drop-in runltp
replacement run on the SUT. It's not the intended use-case (we aim it to be run
on host), but still supported use case.

> > The reason we go to kirk + ltx is that in the future we plan to get rid of
> > runtests, replace them with metadata info (that will allow many things [4],
> > but you were Cc at the discussion [5]). Once this happen, runltp (or any custom
> > script / framework runner) will be broken. But this likely take long time.

> Right, using a vendored runltp would just buy us a small amount of time
> so likely isn't a long term viable solution sadly. I'd consider it if
> it were but it doesn't seem a good investment of time.

I would say adopting kirk in minimalistic way (or identify what you need and
it's still missing) would be sure safer in a long term.

Kind regards,
Petr

> Cheers,

> Richard

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp