Re: [LTP] [doc, runtest] [was: Re: [PATCH] cve: add CVE-2025-38236 test]

[Petr Vorel <pvorel@suse.cz>]

Hi Cyril, all,

> Hi!
> > This problem happen on all runtest files, fixing just one does not fix the
> > problem.

> Well we can do that for any runtest file that has clear definition of
> which tests belongs there. For CVE it's crystal clear, tests that have
> cve tag should be there. For the rest of the runtest files, it's not so
> much. Maybe for syscalls we may be able to do so.

> The main thing is that we have to start somewhere got eventually get
> there. I just quickly looked at the cve runtest file and figured out
> that we have to add tests variants somewhere into the metadata. I.e.
> quite a few of the CVE tests have command line options in the runtest
> file which has to be stored somewhere else.

Thanks for analysis. I put your investigation into an issue:

https://github.com/linux-test-project/ltp/issues/1253

> > Sure, it'd be possible to generate runtest/cve from metadata. Do we really want
> > to implement it? (I can create a ticket). I guess we would use C and ujson to
> > not require json-c or python3 for building LTP.

> Or we can hook it up directly into the metadata parser, instead of
> parsing the resulting JSON we can act on the data while they are in the
> memory. Matching some tags and writing a test name into a file could be
> easily done.

> > I would be more interested to have section "CVE reproducers" in Statistics page [1].
> > While the same tool could be used to do both goals, when only doc page
> > implemented, it could be easily done in python3 (doc/conf.py already parses
> > ltp.json).

> > When we are at Statistics page, also generating list of reproducers (based on
> > kernel fixes) would be also nice. Because this was implemented in the previous
> > asciidoctor implementation. How about having these lists Statistics, where are
> > other tables already (and linking each test to "Test Catalog")?

> > Also I find "Statistics" name confusing. It says nothing about the content. I
> > wonder if people curiously click on the page or just ignore the page (if they
> > don't like math :)). Maybe "Kernel coverage" or something like that would be
> > more informative.

> I would put the list of reproducers and list of CVE reproducers into a
> separate page that would be have "reproducers" in the name.

https://github.com/linux-test-project/ltp/issues/1254

> And statistics is probably okayish name, since coverage may mislead
> people even more. For example we have a lot of tests for a write()
> syscall yet coverage for all the possible write handlers in kernel is
> very poor and not likely to improve.

Fair enough.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp