Re: [LTP] LTP old API conversion

[Andrea Cervesato via ltp <ltp@lists.linux.it>]

Hi Cyril,

> Hi!
> > .. and tests conversion for tests which are smaller than 200 lines of code
> > requires minimal (if no) edit. I will continue to adapt the ltp-convert skill
> > in order to tweak and to improve this process for bigger tests.
> 
> I would disagree here withe the minimal edit. The old tests are quite
> often garbage. So the human in the process should asses if the test is
> actually doing anything useful and guide the machine to implement better
> test in a case that the original wasn't doing anything useful.

That's indeed what I do. To blindly accept everything turns out to produce a lot
of trash, but in general it's way more easy to refactor tests in this way than
before, at least for the smallest tests. With a good set of rules given to the
LLM we can produce good results at the very first try sometimes.

> 
> > ~~ Said so..
> > 
> > .. since this process seems to be quite straight forward, and with the usage
> > of LLM we could easily generate hundreds of patches per month, we don't really
> > want to flood the ML with garbage and to overwhelm who's involved into
> > maintenance review.
> 
> I'm pretty sure that you will get this down to smaller set as long as
> you actually spend some time thinking about what the test does. At least
> for me I'm able to spend 30% time on a test conversion because LLM, but
> you have to spend some time understanding what the test does and making
> sure it makes sense. Otherwise it's garbage in - garbage out.

I counted a x4 times faster conversion compared to the previous manual process,
at least for 100-200 lines tests. Bigger ones will be more challenging and after
a few attempt the conversion time drops significantly.

An another thing is how we want to push these patches in the repository. At the
moment kernel is using some special tags to identify if patches have been
generated by LLMs or by humans. This is something we should think about,
because we already have a few examples recently (i.e. your ulimit01, newuname01
tests and my listns() testing suite) where LLMs where doing most of the job.
Also, we are receiving more and more patches which seem to be written by AI.

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp