Re: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework

Knut Omang
2019-05-09T11:55:48Z

On Wed, 2019-05-08 at 23:20 -0400, Theodore Ts'o wrote:
> On Wed, May 08, 2019 at 07:13:59PM -0700, Frank Rowand wrote:
> > > If you want to use vice grips as a hammer, screwdriver, monkey wrench,
> > > etc. there's nothing stopping you from doing that. But it's not fair
> > > to object to other people who might want to use better tools.
> > >
> > > The reality is that we have a lot of testing tools. It's not just
> > > kselftests. There is xfstests for file system code, blktests for
> > > block layer tests, etc. We use the right tool for the right job.
> >
> > More specious arguments.
>
> Well, *I* don't think they are specious; so I think we're going to
> have to agree to disagree.

Looking at both Frank's and Ted's arguments here, I don't think you
really disagree, I just think you are having different classes of tests in mind.

In my view it's useful to think in terms of two main categories of
interesting unit tests for kernel code (using the term "unit test" pragmatically):

1) Tests that exercises typically algorithmic or intricate, complex
code with relatively few outside dependencies, or where the dependencies
are considered worth mocking, such as the basics of container data
structures or page table code. If I get you right, Ted, the tests
you refer to in this thread are such tests. I believe covering this space
is the goal Brendan has in mind for KUnit.

2) Tests that exercises interaction between a module under test and other
parts of the kernel, such as testing intricacies of the interaction of
a driver or file system with the rest of the kernel, and with hardware,
whether that is real hardware or a model/emulation.
Using your testing needs as example again, Ted, from my shallow understanding,
you have such needs within the context of xfstests (https://github.com/tytso/xfstests)

To 1) I agree with Frank in that the problem with using UML is that you still have to
relate to the complexity of a kernel run time system, while what you really want for these
types of tests is just to compile a couple of kernel source files in a normal user land
context, to allow the use of Valgrind and other user space tools on the code. The
challenge is to get the code compiled in such an environment as it usually relies on
subtle kernel macros and definitions, which is why UML seems like such an attractive
solution. Like Frank I really see no big difference from a testing and debugging
perspective of UML versus running inside a Qemu/KVM process, and I think I have an idea
for a better solution:

In the early phases of the SIF project which mention below, I did a lot of experimentation around this. My biggest challenge then was to test the driver
implementation of the pages table handling of an Intel page table compatible on-device
MMU, using a mix of page sizes, but with a few subtle limitations in the hardware. With some efforts of code generation and heavy automated use of
compiler feedback, I was able
to do that to great satisfaction, as it probably saved the project a lot of time in
debugging, and myself a lot of pain :)

To 2) most of the current xfstests (if not all?) are user space tests that do not use
extra test specific kernel code, or test specific changes to the modules under test (am I
right, Ted?) and I believe that's just as it should be: if something can be exercised well enough from user space, then that's the easier approach.

However sometimes the test cannot be made easily without interacting directly
with internal kernel interfaces, or having such interaction would greatly simplify or
increase the precision of the test. This need was the initial motivation for us to make
KTF (https://github.com/oracle/ktf, http://heim.ifi.uio.no/~knuto/ktf/index.html) which we are working on to adapt to fit naturally and in the right way
as a kernel patch set.

We developed the SIF infiniband HCA driver
(https://github.com/oracle/linux-uek/tree/uek4/qu7/drivers/infiniband/hw/sif)
and associated user level libraries in what I like to call a "pragmatically test driven"
way. At the end of the project we had quite a few unit tests, but only a small fraction of them were KTF tests, most of the testing needs were covered
by user land unit tests,
and higher level application testing.

To you Frank, and your concern about having to learn yet another tool with it's own set of syntax, I completely agree with you. We definitely would want
to minimize the need to
learn new ways, which is why I think it is important to see the whole complex of unit
testing together, and at least make sure it works in a unified and efficient way from a
syntax and operational way.

With KTF we focus on trying to make kernel testing as similar and integrated with user
space tests as possible, using similar test macros, and also to not reinvent more wheels than necessary by basing reporting and test execution on
existing user land tools.
KTF integrates with Googletest for this functionality. This also makes the reporting format discussion here irrelevant for KTF, as KTF supports whatever
reporting format the user land tool supports - Googletest for instance naturally supports pluggable reporting implementations, and there already seems
to be a TAP reporting extension out there (I haven't tried it yet though)

Using and relating to an existing user land framework allows us to have a set of
tests that works the same way from a user/developer perspective,
but some of them are kernel only tests, some are ordinary user land
tests, exercising system call boundaries and other kernel
interfaces, and some are what we call "hybrid", where parts of
the test run in user mode and parts in kernel mode.

I hope we can discuss this complex in more detail, for instance at the testing
and fuzzing workshop at LPC later this year, where I have proposed a topic for it.

Thanks,
Knut