Re: Advancing the testing for the CAN subsystem

[Felix Maurer <fmaurer@redhat.com>]

Hi Oliver, all,

On 21.02.25 09:04, Oliver Hartkopp wrote:
> Hi Felix, all,
> 
> by now I was only aware of syzbot who is sometimes showing up with some
> splats
> 
> and the Linux Test Project (LTP)
> 
> https://github.com/linux-test-project/ltp/tree/master/testcases/network/can
> 
> which e.g. provides CAN filter tests which you can also find in the can-
> tests repo.
> 
> On 20.02.25 14:03, Felix Maurer wrote:
>> Hi Marc, Oliver, and linux-can community,
>>
>> we are reaching out to you because we would like to advance the testing
>> of the kernel CAN subsystem. We, that's Davide, Filippo and I, are
>> volunteering to provide the patches for this, but would like to get your
>> feedback and opinions first.
> 
> I would definitely like it and support it :-)

Great to hear :-)

>> We know about the can-tests repository[1] and think this is a good
>> starting point for our efforts. Currently, there are two main activities
>> we'd like to do:
>>
>> - Promote the test cases in can-tests to become part of the kernel
>> selftests: This would mainly get the tests closer to the upstream kernel
>> development, both in terms of maintenance and actually running them. CI
>> systems like LKFT and CKI could easily be continuously running the
>> tests.
> 
> Ok. Just for my understanding: Bringing the test cases into tools/
> testing/selftests would be the enabler to run LKFT and CKI, right?

It's not strictly necessary to have the test cases in
tools/testing/selftests to run them in LKFT and CKI, but it makes
running them much simpler. The selftests are already built in most
kernel CIs and often automatically run. In contrast, an external repo
would require to define a test case for each CI system.

> Does it have any impact or improvement for syzbot or LTP too, e.g. that
> we can also improve their test quality?

I don't think this would automatically improve syzbot or LTP.

As far as I know, LTP doesn't run the kernel selftests. It would rather
be an alternative place for the tests if they should (for some reason)
be separate from the kernel tree. But imho, bringing the test cases
closer to the kernel is a good thing. It makes sure the test cases match
the kernel version and makes them easy to discover and run for contributors.

syzbot is unrelated to other test cases. It relies on it's own
definitions to fuzz the kernel. For networking, there are two different
approaches to these definitions:

1) the definition of the syscall interfaces for a subsystem, e.g., in
[2] for CAN which I think is where the current sysbot splats are coming
from. New syscalls, options, etc. would need to be added there to extend
coverage, which should be the matter of a pull request. I only took a
quick look so I can't say for sure what might be missing here, but I
didn't see a mention of the cangw netlink interface and ISOTP there.
syzcaller might be able to discover bugs there as well by fuzzing
syscalls in general, but it of course works way better when there is a
definition of the interface available.

2) For all protocols over Ethernet, syzkaller can do external network
fuzzing [3]. With that, the network stack is fuzzed from the "outside"
by sending it packets as if they were coming from another machine,
guided by definitions of the protocols, similar to the syscall
definitions. I think it's a long shot to enable similar testing for CAN
and not something I would aim for in the near future.

[2]:
https://github.com/google/syzkaller/blob/master/sys/linux/socket_can.txt
[3]:
https://github.com/google/syzkaller/blob/master/docs/linux/external_fuzzing_network.md

>> The downside is that existing automation depending on can-tests
>> (which we don't know about) would need to be modified.
> 
> I don't know about any existing automation based in can-tests either.
> The reason for the can-test was mostly to document my own PoCs when
> adding new features.
> 
> I have no objections to stay with this repo as a PoC playground that
> might also help developers to get inspirations - and start something new
> with the current code base which aims to fuel the kernel selftests.

Okay, that's good to know can-tests is mostly PoCs for new features. It
can stay around of course, I know other subsystems have similar repos
(xdp for example). I'd add a note though that the selftests exist for
already merged features.

>> - Extend the coverage of the tests: This could include testing for,
>> e.g., vcan, vxcan, and the cangw netlink interface. But we're open to
>> feedback here if you see any pressing areas.
> 
> I see no pressing areas right now. But the recently integrated CAN XL
> support might get some attention ;-)

Alright, I'll keep that in the back of my head :-)

Thanks,
   Felix