[LTP] LTP old API conversion

[LTP] LTP old API conversion

[Andrea Cervesato via ltp]

Hi all,

so we still have ~200 patches to refactor and to move from old LTP API into
the new LTP API. That would be really useful for tests maintenance and long
term supports, such as the `runtest` removal and replacement with a smarter
tests filtering/grouping.

This is a tedious task that requires a huge amount of work and in the past
years we managed to convert hundreds of tests by hand, each one requiring
multiple reviews iterations.

It was overwhelming not only for developers, but also for reviewers who
were stucked by reviewing new patches + tests rewriting.

In 2026 we have the chance to accelerate this transition from old API to
new API using LLMs and, as we discussed in the yesterday LTP after release
meeting, we might be in the right way to start doing it (at least for
smaller tests).

I created a set of configurations and skills in my personal repo which can
be used to start this process: https://github.com/acerv/agents-config.
It's maily tested using Claude Code, since it's the model which perform
the best (at the moment), but any other model can be used.

I experimented with a list of tests that can be obtained with the following
command:

wc -l $(grep -R '"test\.h"' --include="*.c" testcases/kernel/ | cut -d: -f1) | sort -g

.. and tests conversion for tests which are smaller than 200 lines of code
requires minimal (if no) edit. I will continue to adapt the ltp-convert skill
in order to tweak and to improve this process for bigger tests.

~~ Said so..

.. since this process seems to be quite straight forward, and with the usage
of LLM we could easily generate hundreds of patches per month, we don't really
want to flood the ML with garbage and to overwhelm who's involved into
maintenance review.

How we should organize this job?
Should we set a maximum amount of tests refactoring per month?
How do we organize these patches? (i.e. with blocks of patches)

Kind regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] LTP old API conversion

[Petr Vorel]

Hi Andrea,

[ Cc others who were on LTP meeting + Sebastian who might be interested ]

> Hi all,

> so we still have ~200 patches to refactor and to move from old LTP API into
> the new LTP API. That would be really useful for tests maintenance and long
> term supports, such as the `runtest` removal and replacement with a smarter
> tests filtering/grouping.

> This is a tedious task that requires a huge amount of work and in the past
> years we managed to convert hundreds of tests by hand, each one requiring
> multiple reviews iterations.

> It was overwhelming not only for developers, but also for reviewers who
> were stucked by reviewing new patches + tests rewriting.

> In 2026 we have the chance to accelerate this transition from old API to
> new API using LLMs and, as we discussed in the yesterday LTP after release
> meeting, we might be in the right way to start doing it (at least for
> smaller tests).

> I created a set of configurations and skills in my personal repo which can
> be used to start this process: https://github.com/acerv/agents-config.
> It's maily tested using Claude Code, since it's the model which perform
> the best (at the moment), but any other model can be used.

> I experimented with a list of tests that can be obtained with the following
> command:

> wc -l $(grep -R '"test\.h"' --include="*.c" testcases/kernel/ | cut -d: -f1) | sort -g

wc -l $(grep -R '"test\.h"' --include="*.c" testcases/ | cut -d: -f1) | sort -g

Otherwise you miss 12 old API tests.

> .. and tests conversion for tests which are smaller than 200 lines of code
> requires minimal (if no) edit. I will continue to adapt the ltp-convert skill
> in order to tweak and to improve this process for bigger tests.

> ~~ Said so..

> .. since this process seems to be quite straight forward, and with the usage
> of LLM we could easily generate hundreds of patches per month, we don't really
> want to flood the ML with garbage and to overwhelm who's involved into
> maintenance review.

> How we should organize this job?
> Should we set a maximum amount of tests refactoring per month?
> How do we organize these patches? (i.e. with blocks of patches)

I'd vote for limiting patchsets to max tests in a single directory.
Why? Smaller patchset is easier to review. And if just some of the commits are
accepted then fewer commits need to be rebased.

Kind regards,
Petr

> Kind regards,

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] LTP old API conversion

[Andrea Cervesato via ltp]

Hi Petr,

> I'd vote for limiting patchsets to max tests in a single directory.
> Why? Smaller patchset is easier to review. And if just some of the commits are
> accepted then fewer commits need to be rebased.

Do you mean one patch-set for multiple (unrelated) tests (for instance syscalls
folder) [1], or one patch-set per testing suite [2] ?

[1] would be 20 random tests from testcases/kerne/syscalls
[2] i.e. all tests to refactor in `testcases/kernel/mem/mmapstress`

I would follow the first approach if we want to send a big amount of patches,
or the second if we want to be more specific.

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] LTP old API conversion

[Cyril Hrubis]

Hi!
> .. and tests conversion for tests which are smaller than 200 lines of code
> requires minimal (if no) edit. I will continue to adapt the ltp-convert skill
> in order to tweak and to improve this process for bigger tests.

I would disagree here withe the minimal edit. The old tests are quite
often garbage. So the human in the process should asses if the test is
actually doing anything useful and guide the machine to implement better
test in a case that the original wasn't doing anything useful.

> ~~ Said so..
> 
> .. since this process seems to be quite straight forward, and with the usage
> of LLM we could easily generate hundreds of patches per month, we don't really
> want to flood the ML with garbage and to overwhelm who's involved into
> maintenance review.

I'm pretty sure that you will get this down to smaller set as long as
you actually spend some time thinking about what the test does. At least
for me I'm able to spend 30% time on a test conversion because LLM, but
you have to spend some time understanding what the test does and making
sure it makes sense. Otherwise it's garbage in - garbage out.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] LTP old API conversion

[Andrea Cervesato via ltp]

Hi Cyril,

> Hi!
> > .. and tests conversion for tests which are smaller than 200 lines of code
> > requires minimal (if no) edit. I will continue to adapt the ltp-convert skill
> > in order to tweak and to improve this process for bigger tests.
> 
> I would disagree here withe the minimal edit. The old tests are quite
> often garbage. So the human in the process should asses if the test is
> actually doing anything useful and guide the machine to implement better
> test in a case that the original wasn't doing anything useful.

That's indeed what I do. To blindly accept everything turns out to produce a lot
of trash, but in general it's way more easy to refactor tests in this way than
before, at least for the smallest tests. With a good set of rules given to the
LLM we can produce good results at the very first try sometimes.

> 
> > ~~ Said so..
> > 
> > .. since this process seems to be quite straight forward, and with the usage
> > of LLM we could easily generate hundreds of patches per month, we don't really
> > want to flood the ML with garbage and to overwhelm who's involved into
> > maintenance review.
> 
> I'm pretty sure that you will get this down to smaller set as long as
> you actually spend some time thinking about what the test does. At least
> for me I'm able to spend 30% time on a test conversion because LLM, but
> you have to spend some time understanding what the test does and making
> sure it makes sense. Otherwise it's garbage in - garbage out.

I counted a x4 times faster conversion compared to the previous manual process,
at least for 100-200 lines tests. Bigger ones will be more challenging and after
a few attempt the conversion time drops significantly.

An another thing is how we want to push these patches in the repository. At the
moment kernel is using some special tags to identify if patches have been
generated by LLMs or by humans. This is something we should think about,
because we already have a few examples recently (i.e. your ulimit01, newuname01
tests and my listns() testing suite) where LLMs where doing most of the job.
Also, we are receiving more and more patches which seem to be written by AI.

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] LTP old API conversion

[Petr Vorel]

> Hi Petr,

> > I'd vote for limiting patchsets to max tests in a single directory.
> > Why? Smaller patchset is easier to review. And if just some of the commits are
> > accepted then fewer commits need to be rebased.

> Do you mean one patch-set for multiple (unrelated) tests (for instance syscalls
> folder) [1], or one patch-set per testing suite [2] ?

I meant [2], i.e. tests from certain folder (usually tests related to single syscall).
But it's just a suggestion, feel free to do it differently if it's significantly
simpler.

Thanks for this effort.

Kind regards,
Petr

> [1] would be 20 random tests from testcases/kerne/syscalls
> [2] i.e. all tests to refactor in `testcases/kernel/mem/mmapstress`

> I would follow the first approach if we want to send a big amount of patches,
> or the second if we want to be more specific.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] LTP old API conversion

[Li Wang via ltp]

On Tue, Mar 17, 2026 at 04:32:30PM +0100, Cyril Hrubis wrote:
> Hi!
> > .. and tests conversion for tests which are smaller than 200 lines of code
> > requires minimal (if no) edit. I will continue to adapt the ltp-convert skill
> > in order to tweak and to improve this process for bigger tests.
> 
> I would disagree here withe the minimal edit. The old tests are quite
> often garbage. So the human in the process should asses if the test is
> actually doing anything useful and guide the machine to implement better
> test in a case that the original wasn't doing anything useful.
> 
> > ~~ Said so..
> > 
> > .. since this process seems to be quite straight forward, and with the usage
> > of LLM we could easily generate hundreds of patches per month, we don't really
> > want to flood the ML with garbage and to overwhelm who's involved into
> > maintenance review.
> 
> I'm pretty sure that you will get this down to smaller set as long as
> you actually spend some time thinking about what the test does. At least
> for me I'm able to spend 30% time on a test conversion because LLM, but
> you have to spend some time understanding what the test does and making
> sure it makes sense. Otherwise it's garbage in - garbage out.

Yes but not entirely. We can have AI summarize test methods within
code comments, which helps us grasp the underlying logic much faster
than reading through legacy code directly.

And, nowdays AI is gradually becoming the producer of code, while
humans(specifically maintainers) assume the role of the ultimate
guardians of code quality.

-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] LTP old API conversion

[Li Wang via ltp]

On Tue, Mar 17, 2026 at 11:39:20PM +0100, Petr Vorel wrote:
> > Hi Petr,
> 
> > > I'd vote for limiting patchsets to max tests in a single directory.
> > > Why? Smaller patchset is easier to review. And if just some of the commits are
> > > accepted then fewer commits need to be rebased.
> 
> > Do you mean one patch-set for multiple (unrelated) tests (for instance syscalls
> > folder) [1], or one patch-set per testing suite [2] ?
> 
> I meant [2], i.e. tests from certain folder (usually tests related to single syscall).
> But it's just a suggestion, feel free to do it differently if it's significantly
> simpler.

Some tests may relocate their folder after the refacoring.

Regardless of which method we ultimately adopt, I recommend that
the AI send no more than 10 patches per week [1].

Furthermore, new patches should be sent only after the previous
patchset has been fully processed.

[1] Given that the total number of legacy tests is only 242,
the cleanup work will not take much time.

-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] LTP old API conversion

[Andrea Cervesato via ltp]

> An another thing is how we want to push these patches in the repository. At the
> moment kernel is using some special tags to identify if patches have been
> generated by LLMs or by humans. This is something we should think about,
> because we already have a few examples recently (i.e. your ulimit01, newuname01
> tests and my listns() testing suite) where LLMs where doing most of the job.
> Also, we are receiving more and more patches which seem to be written by AI.

What about this ?

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp