Re: [RFC PATCH 7/7] tools: add skeleton code for userland testing of VMA logic

[Kees Cook <kees@kernel.org>]

On Thu, Jun 27, 2024 at 08:31:07PM +0100, Lorenzo Stoakes wrote:
> On Thu, Jun 27, 2024 at 02:25:36PM -0400, Liam R. Howlett wrote:
> > * Kees Cook <kees@kernel.org> [240627 12:58]:
> > > On Thu, Jun 27, 2024 at 11:39:32AM +0100, Lorenzo Stoakes wrote:
> > > > Establish a new userland VMA unit testing implementation under
> > > > tools/testing which utilises existing logic providing maple tree support in
> > > > userland utilising the now-shared code previously exclusive to radix tree
> > > > testing.
> > > >
> > > > This provides fundamental VMA operations whose API is defined in mm/vma.h,
> > > > while stubbing out superfluous functionality.
> > > >
> > > > This exists as a proof-of-concept, with the test implementation functional
> > > > and sufficient to allow userland compilation of vma.c, but containing only
> > > > cursory tests to demonstrate basic functionality.
> > >
> > > Interesting! Why do you want to have this in userspace instead of just
> > > wiring up what you have here to KUnit so testing can be performed by
> > > existing CI systems that are running all the KUnit tests?
> >
> > The primary reason we did the maple tree testing in userspace was for
> > speed of testing.  We don't need to build the kernel, but a subset of
> > APIs.  Debugging problems is also much quicker since we can instrument
> > and rebuild, iterate down faster.  Tracing every call to the maple tree
> > on boot alone is massive.
> >
> > It is also difficult to verify the vma correctness without exposing APIs
> > we don't want exported (or, I guess, parse proc files..).  On my side, I
> > have a module for testing the overall interface while I have more tests
> > on the userspace side that poke and prod on internal states, and
> > userspace rcu testing is possible.  I expect the same issues on the vma
> > side.
> >
> > Adding tests can also be made very efficient with tracepoints dumping
> > something to add to an array, for example.
> >
> > Finally, you have ultimate control on what other functions return (or
> > do) - so you can fail allocations to test error paths, for example.  Or
> > set the external function to fail after N allocations.  This comes in
> > handy when a syzbot reports a failed allocation at line X caused a
> > crash.
> >
> > This has worked out phenomenally on the maple tree side.  I've been able
> > to record boot failures and import them, syzbot tests, and fuzzer tests.
> > The result is a huge list of tests that allowed me to rewrite my node
> > replacement algorithm and have it just work, once it passed the
> > collected tests.
> >
> > I haven't used kunit as much as I have userspace testing, so I cannot
> > say if all of these points are not possible, but I didn't see a way to
> > test races like I do with rcu in userspace.
> >
> > Thanks,
> > Liam
> 
> Liam's response is excellent, and obviously I agree
> wholeheartedly. Additionally, I'm not really experienced with kunit, but
> surely it's implemented as a kernel module somehow? If so, these interfaces
> are largely not exported so it wouldn't be functional as a unit test.

(KUnit has a method for doing optional exports.)

> And also as Liam says, it'd be very difficult to test this stuff _in_ the
> kernel without unwanted side-effects triggering and it'd be very difficult
> to isolate or mock components we don't want to play a role (for instance -
> rlimits that we might not be able to control).

Yeah, this all makes good sense. Thanks for the details! I've been
looking at ways to test signals and task tree helpers[1] and avoiding
side-effects in the kernel has been quite difficult. I might have to study
your approach here and see if the same could be done for my cases. :)

-Kees

[1] https://lore.kernel.org/lkml/20230814210508.never.871-kees@kernel.org/

-- 
Kees Cook