[LTP] Identify current test coverage and clarify contribution opportunities

[LTP] Identify current test coverage and clarify contribution opportunities

[Luigi Pellecchia]

Hi all,
I'm Luigi Pellecchia, a Principal SW Quality Engineer at Red Hat.
I developed an Open Source Software Quality Management Tool, named "BASIL
The FuSa Spice" that can help the LTP keep track of the test case coverage
against man pages and to clarify contribution opportunities to new members.
I prepared an initial demo I shared on LinkedIn at
https://www.linkedin.com/posts/luigi-pellecchia_how-basil-can-help-linux-test-project-to-activity-7186248090129956864-d-vC?utm_source=share&utm_medium=member_desktop
This tool is under the hood of ELISA (Linux Foundation) github at
https://github.com/elisa-tech/BASIL

Any feedback will be greatly appreciated

Please let me know if you have any questions, or if you'd like to see a
live demo of the tool.

Regards

--
Luigi

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] Identify current test coverage and clarify contribution opportunities

[Andrea Cervesato via ltp]

Hi Luigi,

On 4/17/24 08:32, Luigi Pellecchia wrote:
> Hi all,
> I'm Luigi Pellecchia, a Principal SW Quality Engineer at Red Hat.
> I developed an Open Source Software Quality Management Tool, named "BASIL
> The FuSa Spice" that can help the LTP keep track of the test case coverage
> against man pages and to clarify contribution opportunities to new members.
> I prepared an initial demo I shared on LinkedIn at
> https://www.linkedin.com/posts/luigi-pellecchia_how-basil-can-help-linux-test-project-to-activity-7186248090129956864-d-vC?utm_source=share&utm_medium=member_desktop
Thanks for sharing. I watched the demo and it can be useful in certain 
circumstances, but in others it could be tricky. I will explain myself 
at the end of the email.
> This tool is under the hood of ELISA (Linux Foundation) github at
> https://github.com/elisa-tech/BASIL
>
> Any feedback will be greatly appreciated
>
> Please let me know if you have any questions, or if you'd like to see a
> live demo of the tool.
>
> Regards
>
> --
> Luigi
>
One of the things we noticed is that man pages are not updated too often 
and some kernel features are not covered by the manuals. This would be 
an issue when it comes to calculate tests coverage and some of us think 
that this approach could be hard to follow.

Also, LTP has a lack of men power nowadays (especially when it comes to 
reviews) which means coverage report is good to track LTP status, but we 
would take ages to implement remaining kernel tests. There's an internal 
discussion in LTP about bringing more people inside the project and we 
are trying to move forward to achieve this goal (see new documentation). 
That would help the whole project.

We have a monthly LTP meeting. The next will probably be the 8th of may 
(I need to send the email yet). It's public, so feel free to join if you 
want.

Best regards,
Andrea Cervesato


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] Identify current test coverage and clarify contribution opportunities

[Li Wang]

On Wed, Apr 17, 2024 at 3:43 PM Andrea Cervesato via ltp <ltp@lists.linux.it>
wrote:

> Hi Luigi,
>
> On 4/17/24 08:32, Luigi Pellecchia wrote:
> > Hi all,
> > I'm Luigi Pellecchia, a Principal SW Quality Engineer at Red Hat.
> > I developed an Open Source Software Quality Management Tool, named "BASIL
> > The FuSa Spice" that can help the LTP keep track of the test case
> coverage
> > against man pages and to clarify contribution opportunities to new
> members.
> > I prepared an initial demo I shared on LinkedIn at
> >
> https://www.linkedin.com/posts/luigi-pellecchia_how-basil-can-help-linux-test-project-to-activity-7186248090129956864-d-vC?utm_source=share&utm_medium=member_desktop
> Thanks for sharing. I watched the demo and it can be useful in certain
> circumstances, but in others it could be tricky. I will explain myself
> at the end of the email.
> > This tool is under the hood of ELISA (Linux Foundation) github at
> > https://github.com/elisa-tech/BASIL
> >
> > Any feedback will be greatly appreciated
> >
> > Please let me know if you have any questions, or if you'd like to see a
> > live demo of the tool.
> >
> > Regards
> >
> > --
> > Luigi
> >
> One of the things we noticed is that man pages are not updated too often
> and some kernel features are not covered by the manuals. This would be
> an issue when it comes to calculate tests coverage and some of us think
> that this approach could be hard to follow.
>
> Also, LTP has a lack of men power nowadays (especially when it comes to
> reviews) which means coverage report is good to track LTP status, but we
> would take ages to implement remaining kernel tests. There's an internal
> discussion in LTP about bringing more people inside the project and we
> are trying to move forward to achieve this goal (see new documentation).
> That would help the whole project.
>

So maybe giving a 'coverage report' to LTP is more valuable than the tool
itself?

At least it can reflect something we weren't aware of, we won't follow up
the report one by one dogmatically, but pick up something useful.



>
> We have a monthly LTP meeting. The next will probably be the 8th of may
> (I need to send the email yet). It's public, so feel free to join if you
> want.
>
> Best regards,
> Andrea Cervesato
>
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp
>
>

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] Fwd: Identify current test coverage and clarify contribution opportunities

[Luigi Pellecchia]

Hi everyone,
First of all, thanks for your valuable feedback.
I'm sorry for the late reply, I was travelling.
I understand all your points and concerns.
I sent this video to you pushed by this piece of the project documentation:

https://linux-test-project.readthedocs.io/en/latest/developers/test_case_tutorial.html

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Find an untested System call
<https://linux-test-project.readthedocs.io/en/latest/developers/test_case_tutorial.html#find-an-untested-system-call>

Try to find an untested system call which has a manual page (i.e. man
syscall produces a result). It is a good idea to Git-clone the latest
kernel man-pages repository.

git clone git://git.kernel.org/pub/scm/docs/man-pages/man-pages.git

At the time of writing, the difference between the latest man-pages release
and the HEAD of the repository (usually the latest commit) is well over 100
commits. This represents about 9 weeks of changes. If you are using a
stable Linux distribution, your man-pages package may well be years old. So
as with the kernel, it is best to have the Git repository as a reference.

You could also find a system call with untested parameters or use whatever
it is you are planning to use the LTP fo
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
from the one you as a project are asking people to discover untested
syscall.
This raised a flag in my mind because anytime someone is doing that
analysis for a selected syscall we quickly lose his contribution because
his effort is not tracked anywhere.
BASIL can help keep track of any contribution and an idea can be to start
tracking only new tests to learn ourselves if this can give some benefits
to the project.

Moreover I'm involved in the ELISA (Linux Foundation) community and we are
going to spin up an instance of this tool.
There are several companies involved in this project that are actively
working on Safety Critical applications that can take advantage from such
pedantic analysis of test coverage against man page and it could be a good
opportunity to onboard new contributors.

With that said, I understand that LTP is not only testing syscall and that
you see more value in other tests, like the regression ones.
As in ELISA we are going to start an analysis, we'd like to leverage your
expertise and suggestions to see if there is something in particular that
can give back to you some important information.
I understand that you are looking for input data and not more work to do
because of workforce constraints, so please, guide me in the right
direction.

To reply to Cyril,
yes a manual effort is required to map snippets to test cases.
To keep the mapping stable an idea can be to select a commit hash of the
man page for a selected version of the syscall.
It is also true that if a man page changes, that can be for a good reason
and probably it is better to check the mapping.
The effort of refining the effort is really limited because the User
simplifies this process.

I'm sorry for the nanosleep example. I choose a simple syscall to perform a
quick analysis.

I'll try to join the next meeting, thanks for the invitation.
I hope you see this thread as a constructive proposal and not something
that can deviate your effort from your project goal.

Cheers

--
Luigi

On Wed, Apr 17, 2024 at 1:43 AM Li Wang <liwang@redhat.com> wrote:

> On Wed, Apr 17, 2024 at 3:43 PM Andrea Cervesato via ltp <
> ltp@lists.linux.it>
> wrote:
>
> > Hi Luigi,
> >
> > On 4/17/24 08:32, Luigi Pellecchia wrote:
> > > Hi all,
> > > I'm Luigi Pellecchia, a Principal SW Quality Engineer at Red Hat.
> > > I developed an Open Source Software Quality Management Tool, named
> "BASIL
> > > The FuSa Spice" that can help the LTP keep track of the test case
> > coverage
> > > against man pages and to clarify contribution opportunities to new
> > members.
> > > I prepared an initial demo I shared on LinkedIn at
> > >
> >
> https://www.linkedin.com/posts/luigi-pellecchia_how-basil-can-help-linux-test-project-to-activity-7186248090129956864-d-vC?utm_source=share&utm_medium=member_desktop
> > Thanks for sharing. I watched the demo and it can be useful in certain
> > circumstances, but in others it could be tricky. I will explain myself
> > at the end of the email.
> > > This tool is under the hood of ELISA (Linux Foundation) github at
> > > https://github.com/elisa-tech/BASIL
> > >
> > > Any feedback will be greatly appreciated
> > >
> > > Please let me know if you have any questions, or if you'd like to see a
> > > live demo of the tool.
> > >
> > > Regards
> > >
> > > --
> > > Luigi
> > >
> > One of the things we noticed is that man pages are not updated too often
> > and some kernel features are not covered by the manuals. This would be
> > an issue when it comes to calculate tests coverage and some of us think
> > that this approach could be hard to follow.
> >
> > Also, LTP has a lack of men power nowadays (especially when it comes to
> > reviews) which means coverage report is good to track LTP status, but we
> > would take ages to implement remaining kernel tests. There's an internal
> > discussion in LTP about bringing more people inside the project and we
> > are trying to move forward to achieve this goal (see new documentation).
> > That would help the whole project.
> >
>
> So maybe giving a 'coverage report' to LTP is more valuable than the tool
> itself?
>
> At least it can reflect something we weren't aware of, we won't follow up
> the report one by one dogmatically, but pick up something useful.
>
>
>
> >
> > We have a monthly LTP meeting. The next will probably be the 8th of may
> > (I need to send the email yet). It's public, so feel free to join if you
> > want.
> >
> > Best regards,
> > Andrea Cervesato
> >
> >
> > --
> > Mailing list info: https://lists.linux.it/listinfo/ltp
> >
> >
>
> --
> Regards,
> Li Wang
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp
>

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] Identify current test coverage and clarify contribution opportunities

[Cyril Hrubis]

Hi!
> I'm Luigi Pellecchia, a Principal SW Quality Engineer at Red Hat.
> I developed an Open Source Software Quality Management Tool, named "BASIL
> The FuSa Spice" that can help the LTP keep track of the test case coverage
> against man pages and to clarify contribution opportunities to new members.
> I prepared an initial demo I shared on LinkedIn at
> https://www.linkedin.com/posts/luigi-pellecchia_how-basil-can-help-linux-test-project-to-activity-7186248090129956864-d-vC?utm_source=share&utm_medium=member_desktop
> This tool is under the hood of ELISA (Linux Foundation) github at
> https://github.com/elisa-tech/BASIL
> 
> Any feedback will be greatly appreciated

Sorry to break it to you but this is not going to work at all for a
couple of reasons.

Firstly man pages are not complete enough and majority of the kernel
interfaces are completely undocumented and this is not going to get
fixed anytime soon. So any metric based on man pages is doomed to fail.

Secondly from the demo it looks like there is a major manual effort
required to pair man page snippets with testcases, which needs to be
redone each time any of them changes. There are thousands of tests in
LTP, going over them would take years of manpower, that is better spend
elsewhere. We have very obvious gaps in coverage so writing new tests
for subsystem that are sparsely covered is way better than trying to
identify minor coverage gaps in existing tests.

Thirdly writing tests to cover API specification is not exactly the best
strategy, it has been tried before and it didn't produce resonable
results. That may work for very simple libraries but for anything more
complex the reality is more tricky and useful tests often require clever
thinking. The prime example of this is the open posix testsuite inside
LTP where they tried to write tests for each assertion from POSIX. That
often lead to nonsensical tests and we are stil trying to clean up the
fallout from that. Also if you look at any kernel regression tests,
which are the most useful ones, the code does not follow any assertions
from man pages, it usuall does wild stuff that is not documented
anywhere. The most useful tests we have were written with thinking
outside of the box, which is not something you can achieve when trying
to adhere pedantically to a specification.

Also if you look at the example from your presentation, you pointed out
that nanosleep() is not tested againts EFAULT, which is not really
useful to be honest. Sure we should add that testcase, but in 99% of the
cases the userspace buffers are copied to kernel by a common function.
That means that it's very unlikely that we wouldn't catch a problem in
that function since we have thousands of tests that actually check for
EFAULT handling in syscalls. Do you see how pedantic comparsion of
manual pages against tests can easily lead you to something that is
not that useful?

To sum it up, this does not look very useful and has potential to divert
manpower from where it's needed most i.e. actuall test writing.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp